PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs

Ashutosh Hathidara, Sai Shruthi Sistla, Sebastian Schreiber, Sahil Bansal

arXiv:2606.12451v1cs.AIcs.IRcs.LG

TL;DR

Standard tool-retrieval benchmarks may not reveal whether models understand tools or generalize beyond verbose, fully specified queries. ToolSense generates realistic retrieval and factual-probing benchmarks, showing that models strong on standard benchmarks can collapse on realistic queries while retrieval fine-tuning erodes previously acquired tool knowledge.

  • Problem

    Evidence is limited on whether parametric tool retrievers understand specialized tools and perform reliably beyond verbose, fully specified benchmark queries.

  • Method

    ToolSense automatically generates realistic retrieval, multiple-choice, and question-answering benchmarks from any tool catalog to diagnose parametric tool retrieval.

  • Results

    Configurations that perform strongly on standard benchmarks collapse on realistic queries, while Stage 2 retrieval training nearly universally destroys Stage 1 tool knowledge.

  • Takeaways & Limitations

    High constrained recall on verbose in-distribution queries is a poor proxy for real-world capability, and retrieval systems should be evaluated for realistic generalization and knowledge retention.

  • Takeaways & Limitations

    The empirical study focuses on ToolGen’s two-stage paradigm applied to ToolBench, leaving other catalog-agnostic parametric designs for future diagnosis.

Abstract

from arXiv · show

Large language models deployed as agents over large tool catalogs face a critical tool-retrieval bottleneck. As embedding-based retrieval approaches rely on compact encoders that may under-capture specialized tool semantics, parametric tool retrieval addresses this by encoding each tool as a virtual token appended to the LLM vocabulary, fine-tuned in two stages (memorization then retrieval SFT) to use the LLM as a retriever, achieving strong performance on standard ToolBench retrieval benchmarks. Yet these benchmarks use verbose, fully-specified queries, and their evaluation applies constrained decoding that restricts outputs to valid token paths, neither reveals whether the model actually understands its tools. We introduce \textbf{ToolSense}, an open-source LLM-powered diagnostic framework that takes any tool catalog as input and automatically generates three benchmarks: a Realistic Retrieval Benchmark (RRB) with queries at three ambiguity tiers, an MCQ probing benchmark, and a QA probing benchmark. Applying ToolSense to ToolBench (~47k tools) and evaluating five parametric model training configurations reveals a knowledge-retrieval dissociation: on RRB queries, several configurations collapse by ~50-64 percentage points compared to fully-specified ToolBench benchmarks, falling below the embedding-model baseline. Additionally, despite strong retrieval performance, some models score near-random on factual probes, suggesting a knowledge-retrieval dissociation. We open-source the ToolSense framework and the ToolBench diagnostic benchmarks at https://github.com/SAP/toolsense.

1 Introduction

ToolSense diagnoses whether parametric tool retrieval reflects genuine tool knowledge or surface-level query-to-token matching. Applied to ToolBench, it exposes failures on realistic queries and factual probes that standard constrained benchmarks can mask.

  • Diagnostic gap: Standard ToolBench evaluations use verbose, fully specified queries and constrained decoding, potentially masking whether models genuinely recall tool tokens.Constrained decoding restricts generation to valid trie paths, whereas downstream agentic fine-tuning typically requires free-form token generation.
  • ToolSense framework: ToolSense automatically generates realistic retrieval, multiple-choice, and question-answering benchmarks from any tool catalog.The Realistic Retrieval Benchmark uses three ambiguity tiers; MCQ probes discriminative factual knowledge, while QA probes inferential factual knowledge about tool properties.
  • Empirical findings: Configurations that perform strongly on standard benchmarks collapse on realistic queries, falling below nonparametric baselines.This gap indicates that distribution-matched benchmark performance may not transfer to differently phrased user queries.
  • Empirical findings: Stage 2 retrieval training nearly universally destroys the tool knowledge acquired during Stage 1 memorization.Stage 2 is responsible for retrieval gains, while factual probing reveals the associated loss of tool knowledge.

2 Background and Related Work

Prior work develops generative and parametric retrieval for documents, entities, corpora, and tool catalogs, typically evaluated with constrained decoding. ToolSense extends probing and interpretability methods to fine-tuned virtual tokens, testing whether they encode tool semantics rather than task-specific retrieval pointers.

  • Generative and parametric retrieval: Generative retrieval encodes document IDs into transformer parameters and retrieves them through constrained beam search, extending from entities to dense document corpora.DSI introduced document-ID encoding; subsequent work applied autoregressive retrieval to knowledge-grounded generation and dense corpora.
  • Generative and parametric retrieval: Parametric tool retrieval adapts this paradigm to tool catalogs using virtual tokens and two-stage training, while existing systems evaluate exclusively with constrained decoding.The free-form IS protocol is presented as a diagnostic practice for this class of systems.
  • Tool learning in LLMs: ToolBench covers ∼16k real-world APIs (∼47k tools), while related work studies tool-call generation, self-insertion, or retrieval augmentation rather than parametric retrieval semantics.The paper focuses specifically on whether the retrieval mechanism encodes tool semantics, independently of downstream task performance.
  • Probing and interpretability: MCQ and QA probes extend factual-knowledge and representation-probing traditions to virtual tokens learned during fine-tuning.The probes test whether fine-tuning creates genuinely semantic representations or only task-relevant pointers.

3 The ToolSense Framework

ToolSense takes a tool catalog and automatically generates three diagnostic benchmarks: RRB, MCQ, and QA. Its pipelines generate and validate ambiguity-tiered retrieval queries and tool-factual probes while constraining evaluation outputs.

  • Framework overview: ToolSense takes a tool catalog as input and automatically generates three diagnostic benchmark datasets: DRRB, DMCQ, and DQA.Figure 1 names the outputs as DRRB, DMCQ, and DQA.
  • Realistic Retrieval Benchmark: RRB presents short, intent-focused requests at easy, medium, and hard ambiguity levels to test retrieval beyond verbose-style queries.Easy queries target one tool, medium requests span 2–3 tools, and hard goals span 4 or more tools.
  • Realistic Retrieval Benchmark: RRB generation samples domain-stratified anchor tools, constructs cosine-similarity hard-negative pools, generates tiered query-answer pairs, and applies dual validation.Validation combines programmable grounding and name-exclusion checks with an LLM judge assessing naturalness, tier compliance, and label correctness.
  • Probing benchmarks: Both probing benchmarks sample tools, generate and judge question-answer pairs, filter ambiguous items, and refer to each tool only as “this tool”.This forces evaluation from the virtual token vτ rather than the tool name.
  • Probing benchmarks: MCQ generates factual questions with one correct answer and three plausible distractors, while QA generates balanced Yes/No questions about specific, verifiable tool properties.Constrained decoding restricts MCQ outputs to A/B/C/D and QA outputs to Yes/No.

4 Experimental Setup

ToolSense is instantiated on ToolBench to diagnose parametric tool retrieval across generalization, decoding dependence, and semantic tool knowledge. The setup combines two-stage training, diagnostic benchmarks, controlled model configurations, retrieval baselines, and validated evaluation procedures.

  • Experimental scope: ToolSense diagnoses OOD generalization, constrained-decoding dependence, and correspondence between retrieval performance and semantic tool knowledge on ToolBench.All models follow two-stage ToolGen training: memorization maps tool metadata to virtual tokens, while retrieval maps queries to virtual tokens.
  • Data: 46,980 RapidAPI tools and ∼195k verbose query–tool pairs support memorization and retrieval training, with G1/G2/G3 standard evaluation splits totaling 1,092 queries.The standard splits contain 593 G1, 399 G2, and 100 G3 queries and match the verbose training distribution.
  • Diagnostic benchmarks: ToolSense generates a 500-query RRB, a 496-sample MCQ probe, and a 500-item QA probe to evaluate retrieval generalization and semantic understanding orthogonally.RRB difficulty tiers contain 167 easy, 167 medium, and 166 hard queries; MCQ tests discriminative factual knowledge, while QA tests inferential knowledge.
  • Model configurations: Five configurations isolate token format, memorization format, and training method, using flat or hierarchical tokens, single- or multi-format memorization, and FFT or LoRA.TG replicates the original ToolGen setup and serves as the baseline; Stage 2 query–token data are otherwise identical across configurations.
  • Evaluation: Evaluation uses Rc@k, Rf@k, and IS@k at beam width B=50, exact-match MCQ/QA accuracy with constrained decoding, BM25 and te3l baselines, and human agreement validation.Random baselines are 25% for MCQ and 50% for QA; Fleiss’ κ is 1.000 for MCQ, 0.973 for QA, and 0.805 for RRB.

5 Results

ToolSense reveals a knowledge–retrieval dissociation: retrieval SFT raises fully specified-query performance but yields much weaker generalization, trie-dependent generation, and near-random factual knowledge. Flat token formats and LoRA or multi-format training improve robustness, while the dissociation persists across architectures and training-data distributions.

  • Retrieval generalization: Stage 2 lifts G1 retrieval to 90–96%, but RRB gains are modest and TG-5FM (LoRA) regresses from 41.0% to 31.9%.Most configurations gain only 2–18 percentage points over Stage 1 on RRB.
  • Retrieval generalization: 44.4%: TG-3FM is the best parametric RRB model but trails te3l at 55.6%, while BM25 reaches 32.4%.Both retrieval baselines improve from G1 to RRB: BM25 rises from 27.8% to 32.4%, and te3l from 47.0% to 55.6%.
  • Retrieval generalization: RRB performance declines gradually across G2 and G3, which fall between G1 and RRB rather than forming a sharp boundary.The intermediate splits trace an out-of-distribution gradient.
  • Constrained decoding dependence: 0.75–0.85: flat configurations achieve substantially higher RRB internalization than hierarchical configurations at 0.33–0.79, with Cohen’s d = 1.45.Flat tokens maintain or improve IS on RRB, whereas hierarchical tokens decline, indicating greater trie dependence under distribution shift.
  • Semantic tool knowledge: 31.4%: TG reaches ∼95% G1 Rc@50 but only 31.4% 4-way MCQ accuracy and exactly 50.0% binary QA accuracy.The corresponding random baselines are 25% for MCQ and 50% for QA.
  • Training and representation factors: 41.7%: multi-format TG-3FM with LoRA achieves the highest Stage 2 MCQ accuracy across architectures, including 74.2% on Qwen3.5-4B and 76.4% on Gemma3-12B.LoRA also preserves more Gemma3-4B TG-3FM knowledge than FFT: 41.7% versus 29.8% MCQ accuracy.

6 Conclusion

ToolSense automatically generates RRB, MCQ, and QA benchmarks from any tool catalog to diagnose parametric retrieval systems across OOD generalization, trie-dependency, and factual knowledge retention. On ToolBench, it shows that high constrained recall on verbose in-distribution queries is a poor proxy for real-world capability.

  • 6 Conclusion: ToolSense generates three benchmarks—RRB, MCQ, and QA—from any tool catalog.These benchmarks probe parametric retrieval systems along orthogonal axes: OOD generalization, trie-dependency, and factual knowledge retention.
  • 6 Conclusion: High constrained recall on verbose in-distribution queries is a poor proxy for real-world capability.Applying ToolSense to ToolBench reveals this limitation directly.
  • 6 Conclusion: Performance collapses on realistic queries, while hierarchical tokens exhibit deep trie-dependency.These findings expose limitations that standard constrained-recall evaluation does not capture.

Limitations

ToolSense’s empirical diagnosis is limited to ToolGen’s two-stage paradigm on ToolBench, while its LLM-generated benchmarks have only been human-validated on 100 samples each. Broader evaluation across catalog-agnostic parametric designs and larger human studies would strengthen confidence, but scaling annotation is costly.

  • Scope: The empirical study focuses on ToolGen’s two-stage paradigm applied to ToolBench, leaving other catalog-agnostic parametric designs for future diagnosis.The benchmarks are designed to generalize to other parametric retrieval designs.
  • Benchmark validation: 100 samples per benchmark were human-validated with κ ≥0.805, but scaling validation further would strengthen confidence and incur non-trivial annotation cost.The RRB, MCQ, and QA benchmarks are LLM-generated.

Ethical Considerations … LoRA Configuration

The paper reports ethically compliant experiments and details a staged parametric-tool training design spanning realistic evaluation, discriminative memorization, retrieval training, configuration controls, and LoRA implementation. These choices target semantic tool disambiguation and knowledge retention while contrasting verbose ToolBench queries with realistic user requests.

  • Ethical Considerations: Experiments followed the ACL Ethics Policy and relevant research-integrity guidelines, with no known unresolved ethical risks.
  • A ToolBench Standard Splits vs. RRB: Query Style Comparison: RRB queries are concise, intent-focused natural-language requests, unlike verbose ToolBench prompts that paraphrase API documentation and permit surface-level matching.Examples include requests for pre-game odds, branded short links, and streaming options.
  • B Multi-Choice Tool Selection (MCTS) Memorization Dataset: MCTS trains models to select the correct virtual token among K hard negatives, forcing fine-grained semantic discrimination rather than memorized association.Hard negatives come from embedding-based similarity search over the full catalog.
  • B Multi-Choice Tool Selection (MCTS) Memorization Dataset: MCTS produces discriminative token geometry that is more resistant to Stage 2 overwriting, with higher Stage 2 MCQ scores for TG-3FM and TG-5FM.
  • C.1 Stage 1: Memorization Formats: Stage 1 uses five memorization formats, including desc→tok, tok→desc, hierarchical stepwise objectives, and MCTS, to associate tools with virtual tokens from complementary directions.The standard desc→tok format is used across all configurations, while additional hierarchical objectives apply to TG-5FM.
  • C.2 Stage 2: Retrieval Training Data: Stage 2 uses identical ToolBench query-to-token data across all five configurations, differing only in whether the task-specific system prompt is included.
  • C.3 Design Rationale per Configuration: Configuration comparisons isolate system prompts, reverse and discriminative objectives, hierarchical tokens, expanded memorization formats, and FFT versus LoRA parameter-update constraints.FFT versus LoRA specifically tests knowledge retention across sequential training stages.
  • D Training Hyperparameters: Training used 8 NVIDIA H200 GPUs, bf16 precision, maximum sequence length 1024, and effective batch size 1024; LoRA used r=64, α=128, zero dropout, and trainable embeddings.LoRA adapters covered all listed linear projection layers, preserved weight tying, and Stage 2 initialized from the merged Stage 1 LoRA checkpoint.

E Realistic-Query Stage 2 Training · F Internalization Score: Rationale and Degenerate-Case Handling

Training TG on realistic RRB-style queries largely restores realistic-query retrieval but sacrifices verbose-query recall, showing distribution-specific specialization. The Internalization Score (IS) measures unconstrained generation relative to constrained recall, with explicit handling for zero-recall and low-recall cases.

  • E Realistic-Query Stage 2 Training: 284,567 training pairs were generated from realistic queries covering all ∼47k ToolBench tools, while the 500-query RRB evaluation set remained disjoint.Multi-tool samples were expanded into one pair per correct tool, with all other Stage 2 hyperparameters unchanged.
  • E Realistic-Query Stage 2 Training: +44.0pp RRB Rc@50 improvement (43.8%→87.8%) shows TG can handle realistic queries when trained on RRB-style data.This result compares TG trained with RRB-style Stage 2 data against standard Stage 2 training.
  • E Realistic-Query Stage 2 Training: 9.1pp lower G1 verbose recall (95.7%→86.6%) confirms that realistic and verbose queries induce complementary, non-overlapping retrieval behaviors.The model specializes in the query distribution used for Stage 2 training.
  • F Internalization Score: Rationale and Degenerate-Case Handling: IS is conditional: it measures how much knowledge already acquired through Rc@k is internalized in the weights rather than delegated to the trie.Rc@k measures the model’s acquired tool knowledge, whereas IS measures its unconstrained accessibility.
  • F Internalization Score: Rationale and Degenerate-Case Handling: IS must be interpreted alongside Rc: IS = 1.0 can occur at Rc@50 = 0.10 and Rf@50 = 0.10, despite low overall tool knowledge.The example shows that perfect internalization of a small acquired subset does not imply broad learning.
  • F Internalization Score: Rationale and Degenerate-Case Handling: IS = 0 when Rc@k = 0, because a model unable to recall any correct tool with trie guidance has nothing to internalize.This convention prevents spuriously high IS values for uniformly broken models.
  • F Internalization Score: Rationale and Degenerate-Case Handling: The ratio-based IS distinguishes failure modes that the difference ∆@k = Rf@k − Rc@k conflates, including IS = 0.89 at Rc = 0.90 versus IS = 0 at Rc = 0.10.Both examples have ∆ = −0.10, but the first is mostly internalized and the second is entirely trie-dependent.

G Internalization Score: Full Results with Confidence Intervals

The section reports IS@50 with 95% bootstrap confidence intervals across four splits and identifies consistent differences between flat and hierarchical token configurations. Flat models generally perform strongly, whereas hierarchical models incur a pronounced RRB penalty, including for Gemma3-4B and Qwen3.5-4B.

  • Confidence intervals: IS@50 is reported with 95% bootstrap confidence intervals across all four evaluation splits.The intervals are derived by the delta method from bootstrap intervals for Rf@50 and Rc@50, then clipped to [0, 1].
  • Configuration patterns: Flat configurations typically achieve IS>0.65 on both G1 and RRB, while LoRA variants systematically reach ≥0.80.Tables separate flat-token configurations from hierarchical-token configurations with a horizontal rule.
  • Configuration patterns: 0.33 is TG-H’s IS on RRB for Gemma3-4B, versus 0.42 on G1, while Qwen3.5-4B reaches 0.28 on RRB.These results exemplify the pronounced RRB penalty observed for hierarchical configurations across architectures and sizes.
  • Configuration patterns: G3 IS values are frequently 1.00 or near-1.00 for flat configurations.The passage associates this pattern with out-of-distribution single-API queries, though the supplied text is truncated before completing the interpretation.

H Cross-Architecture Probing Results

Cross-architecture results show that larger base models begin with substantially stronger parametric tool knowledge and generally preserve it after retrieval fine-tuning. Knowledge destruction is concentrated in smaller models with full-parameter updates, while LoRA provides the strongest retention for Qwen3.5-4B.

  • Cross-architecture knowledge retention: 55.4% to 31.4%: Gemma3-4B FFT TG suffers a severe MCQ drop, unlike Qwen3.5-4B and Gemma3-12B configurations that maintain or slightly improve MCQ scores.The results attribute greater knowledge destruction to smaller models fine-tuned with full parameter updates.
  • Fine-tuning strategy: 74.2% MCQ: Qwen3.5-4B TG-3FM LoRA retains this score through Stage 2, making LoRA the safest fine-tuning strategy when knowledge retention is required.Hierarchical tokenization remains the most destructive configuration across architectures, while LoRA consistently provides the best knowledge preservation for Qwen3.5-4B.

I Virtual Token Embedding Drift Analysis … P Use of AI Assistants

The paper finds that Stage 2 selectively reshapes virtual-token positions for routing rather than preserving or improving semantic tool knowledge, while ToolSense supplies synthetic retrieval and probing benchmarks plus released artifacts and documentation. Its evaluation materials specify tiered benchmark construction, training prompts, compute, annotation procedures, and AI-assisted development.

  • I Virtual Token Embedding Drift Analysis: 22.9× drift under LoRA reflects direct output-loss gradients on virtual tokens versus negligible input-mediated gradients on base-vocabulary tokens.Base-vocabulary drift is Drel(RAND)=0.0015, described as effectively numerical noise.
  • I Virtual Token Embedding Drift Analysis: 7–24× less Stage 2 repositioning for hierarchical tokens, alongside |∆cosim| < 0.002, indicates near-isotropic geometry without representation collapse.TG-3FM flat has drel=0.024, whereas TG-5FM hier has drel=0.003; nevertheless, hierarchical tokens fail MCQ probes as well.
  • J.1 Realistic Retrieval Benchmark (RRB) Generation Pipeline; K.3 Benchmark Generation Prompts: RRB generation retrieves 13 hard negatives per anchor, creating 14-candidate pools and easy, medium, and hard tiers with progressively broader ambiguity and tool coverage.Easy queries target exactly 1 tool, medium queries target 2–3 tools, and hard queries span 4+ tools.
  • J. Benchmark Generation Details; J.2 MCQ and QA Probing Benchmark Generation; K.3 Benchmark Generation Prompts: MCQ and QA probing use generate-then-judge validation and contain 496 and 500 final items, respectively, testing verifiable tool properties through placeholders.MCQ uses four-way choices with three distractors; QA uses binary yes/no questions with balanced target labels.
  • K System Prompts: Training prompts include desc→tok for all configurations, bidirectional tok→desc and MCTS for TG-3FM/TG-5FM, and hierarchical auxiliary targets only for TG-5FM.Stage 2 retrieval uses a shared natural-language query-to-token prompt except for the no-prompt TG baseline.
  • L Artifact Licenses and Intended Use; M Dataset Documentation: The released ToolSense framework and RRB, MCQ, and QA benchmarks use Apache 2.0, derive from a catalog of ∼47k tools across ∼16k services, and are intended for research rather than unvalidated production decisions.ToolBench raw metadata is not redistributed.
  • N Computational Resources and Software; O Human Annotation Study; P Use of AI Assistants: Experiments used one GPU per model configuration on an 8-NVIDIA-H200 node, while three expert annotators independently validated stratified samples and Claude Code assisted code prototyping and paper presentation.The annotation study collected no personal data and was classified as expert evaluation rather than human-subjects research.
Loading 2606.12451v1…