PaperlayerView exact revision on arXiv

Source-linked AI summary

Cartograph: Federated Tool Discovery with Operator-Attested Retrieval for AI Agents

Justice Owusu Agyemang, Michael Agyare, Kwame Opuni-Boachie Obour Agyekum, Kwame Agyeman-Prempeh Agyekum, Francisca Adoma Acheampong, Jerry John Kponyo

arXiv:2609.30293v1cs.CLcs.AI

TL;DR

Large MCP catalogs make exposing every tool definition costly, and publisher-written descriptions pose a trust challenge for retrieval. Cartograph uses operator-attested cards, two-stage retrieval, and Rift to surface a shortlist; on its evaluated corpus, it reduces tokens and improves R@5 over Jaccard keyword retrieval.

  • Problem

    As connected MCP catalogs grow, deciding which detailed tool definitions to surface is difficult because returning the complete catalog exposes costly schemas before a request.

    Source: §1 · p. 2

  • Method

    Cartograph combines operator-attested, signed capability cards with server-first, then tool-level retrieval and Rift confusable-cluster analysis.

    Source: §7 · p. 15

  • Results

    98.8% fewer input tokens under the stated full-catalog accounting (38,250 →475 per task cycle), with R@5 = 0.816 versus 0.592 for Jaccard keyword retrieval on the 49-query benchmark.

    Source: §7 · p. 15

  • Takeaways & Limitations

    Cartograph controls which tool descriptions are surfaced and records their provenance, complementing code execution approaches that handle tool invocation and processing.

    Source: §7 · p. 15

    The 49-query benchmark is author-constructed and of moderate size; independently sourced queries and a larger benchmark are needed to improve generalizability.

    Source: §6.4 · p. 14

Abstract

from arXiv · show

The Model Context Protocol (MCP) enables AI agents to discover and call tools, but loading every definition becomes expensive as connected catalogs grow. We present Cartograph, a federated MCP proxy that changes agent-visible tool discovery from $O(n)$ catalog traversal to $O(k)$ progressive disclosure. Cartograph combines three mechanisms: (1) operator-attested capability cards, Ed25519-signed descriptions generated under the deploying operator's control rather than ranked publisher copy; (2) Rift, a three-layer confusable-cluster analysis comprising density clustering, query-margin analysis, and token diagnosis; and (3) two-stage retrieval, which ranks servers before tools. On a 22-server, 374-tool deployment, Cartograph exposes three proxy tools instead of 374 definitions. A 49-query author-constructed benchmark yields R@5 of 0.816, compared with 0.592 for a Jaccard keyword baseline, while a measured top-5 discovery exchange uses 475 tokens rather than 42,450 under the stated full-catalog accounting. Rift identifies 49 confusable clusters, including four HIGH-risk clusters in bootstrap-generated cards. An exploratory comparison of 119 LLM-generated descriptions removes the observed zero-distance cluster but shows that mixing card-generation regimes can reduce R@5. Gateway measurements over ten trials add 5ms mean latency (0.8%) relative to direct stdio MCP calls. Cartograph is complementary to code-execution approaches: it controls which tool descriptions are surfaced and records the provenance of the descriptions used for ranking for each query.

1 Introduction

Large federated MCP catalogs make complete tool-definition disclosure costly, while publisher-authored descriptions create a retrieval-trust problem. Cartograph uses operator-attested cards and ranking to surface a small, task-relevant shortlist.

1 Introduction

  • 38,250 tokens of JSON Schema definitions are exposed before a request when all 374 tools across 22 servers are returned.Cartograph is positioned as complementary to code execution with MCP, adding proxy-level retrieval and provenance for selecting definitions.Source: §1 · p. 2
  • No mechanism verifies that publisher self-descriptions accurately represent tools, allowing descriptions to be engineered to win retrieval for inappropriate or dangerous tasks.The cited A2A protocol addresses agent-to-agent discovery and delegation, not publisher trust in tool-level retrieval.Source: §1 · p. 2
  • Cartograph exposes three proxy tools and returns top-k schemas matched to a task using operator-generated, Ed25519-signed capability cards.The agent sends a natural-language task description for retrieval rather than loading all n tool definitions.Source: §1 · p. 2
  • Cartograph is evaluated as the sole MCP proxy across 22 production servers on token efficiency, retrieval quality, confusable-cluster detection, and Gateway overhead.The paper reports R@5 = 0.816 on a 49-query author-constructed benchmark, 49 detected clusters, and 0.8% Gateway overhead versus direct MCP.Source: §1 · p. 2

2 Related Work

Related work spans MCP and agent protocols, tool-use research, dense retrieval, and security. Cartograph is framed as deployment infrastructure for governing the tool descriptions surfaced to agents, with a narrower scope than general security controls.

2.1 MCP Ecosystem and Agent Protocols

  • Cartograph complements MCP and code-execution approaches by treating ranked descriptions as a separately governed artifact that operators attest to independently of publisher descriptions.A2A Agent Cards describe agent capabilities for inter-agent discovery; Cartograph cards describe tool capabilities for trustworthy retrieval and carry operator attestation and cryptographic signatures.Source: §2.1 · p. 3

2.2 Tool Selection and Function Calling

  • Toolformer, Gorilla, ToolLLM, StableToolBench, and the Berkeley Function Calling Leaderboard focus on tool-use capability or evaluation, while Cartograph focuses on infrastructure selecting and governing model-supplied descriptions.The cited work includes Toolformer, Gorilla [13], ToolLLM, StableToolBench, and the Berkeley Function Calling Leaderboard.Source: §2.2 · p. 3

2.3 Dense Retrieval and Embedding Quality

  • Rift applies dense-retrieval evaluation insights to tool discovery by diagnosing predictable, domain-specific failures in tool-summary embeddings.The passage relates those insights to DPR, SPLADE [3], BEIR, and MTEB.Source: §2.3 · p. 3

2.4 Trust and Security in Agent Systems

  • Cartograph records an operator signature over descriptions used for retrieval, but does not establish that the underlying tool is benign or replace registry vetting, sandboxing, or transparency services.The passage contrasts this deployment-level control with tool sandboxing and SCITT.Source: §2.4 · p. 3

3 System Design

Cartograph replaces complete tool disclosure with a ranked shortlist and a fixed proxy interface. Its design combines operator-attested cards, server-first retrieval, confusable-cluster analysis, and a one-shot Gateway.

3.1 Problem Formalization

  • The standard MCP model exposes descriptions and argument schemas for all tools, with context-token cost growing as O(n); Cartograph returns at most k ranked tools, yielding a fixed proxy cost plus a small retrieval budget.The fixed cost covers three discovery tools, measured at 169 tokens; the internal server-centroid ranker is not exposed in agent context.Source: §3.1 · pp. 4–5

3.2 Capability Cards

  • A capability card combines server identity, operator-generated capability and tool descriptions, provenance metadata, and an Ed25519 signature.The operator generates semantic fields rather than copying publisher descriptions verbatim; signatures cover the canonicalized card body.Source: §3.2 · pp. 5–6

3.3 Two-Stage Retrieval

  • Two-stage retrieval ranks server centroids before ranking tools within selected servers, reducing scoring to O(m·d+nS′·d) rather than O(n·d).The candidate-server count adapts to catalog sparsity; the index prioritizes recall@k because a missed tool can fail a task while an extra tool costs a few tokens.Source: §3.3 · p. 6

3.4 Rift: Confusable-Cluster Detection

  • Rift clusters tools by pairwise cosine distance, tests synthetic argument-derived queries for retrieval margins, and diagnoses shared versus distinguishing summary tokens.A negative margin means a wrong tool outranks the correct one; tools with no unique lexical signal receive a REGENERATE diagnosis.Source: §3.4 · p. 7

3.5 Gateway

  • The Gateway invokes tools through a one-shot spawn, connect, invoke, and teardown sequence, trading statelessness for per-call spawn overhead.Its latency decomposition distinguishes subprocess startup, MCP handshake, tool execution, and wrapper overhead.Source: §3.5 · pp. 7–8

4 Implementation

Cartograph is implemented as a Python MCP proxy organized around cards, retrieval, Rift, and Gateway subsystems. Bootstrap generates and signs cards from enabled servers before building the index.

4 Implementation

  • The Python 3.11 MCP server exposes discover, status, and call tools for retrieval, catalog overview, and Gateway invocation.The implementation spans approximately 1,800 lines across 12 modules.Source: §4 · p. 8
  • The module architecture separates card models and signing, two-stage indexing, card generation, Rift analysis, stub generation, Gateway adapters, configuration, resolution, and the proxy server.Source: §4 · p. 8
  • Bootstrap connects to enabled stdio servers, obtains tools/list, generates and signs cards, and builds an index over 22 servers and 374 tools.Cards can use a configured DeepSeek model or a deterministic bootstrap template; normal startup loads available cards without rewriting them.Source: §4 · p. 8

5 Evaluation

Cartograph is evaluated across token efficiency, retrieval quality, confusable-cluster detection, LLM-generated cards, and Gateway overhead. The evaluation uses a 49-query author-constructed benchmark and a 22-server, 374-tool corpus.

5.1 Token Efficiency

  • 475 tokens per task cycle for Cartograph compares with 38,250 tokens of JSON Schema definitions under the stated full-catalog accounting.The 475-token total comprises a 169-token fixed tools/list response and approximately 306 tokens for the top five tools; the reasoning-cost estimate varies by task and model.Source: §5.1 · pp. 8–9, Table 2 · p. 9

5.2 Retrieval Quality

  • R@5 = 0.816 on the 49-query author-constructed benchmark, with results reported as descriptive rather than population-level claims.The benchmark covers 13 of 22 servers; the authors note it is moderate in size and lacks a preregistered statistical test.Source: §5.2 · pp. 9–10, Table 3 · p. 9

5.3 Rift: Confusable-Cluster Detection

  • 49 confusable clusters affecting 136 tools were detected by Rift, including 4 HIGH-, 8 MEDIUM-, and 37 LOW-risk clusters.Within-server collisions account for 91.8% of collisions; margin analysis assigns risk labels, while density clustering finds candidates and token diagnosis provides explanations.Source: §5.3 · pp. 10–11, Table 4 · p. 11

5.4 LLM-Generated Cards

  • 119 LLM-generated cards had distinct descriptions, eliminating the observed degenerate cluster of 29 identical bootstrap descriptions.The comparison covers seven servers and is exploratory because the complete catalog does not use a consistent description regime.Source: §5.3 · p. 10, §5.4 · pp. 11–12, Table 6 · p. 12
  • The hybrid index raises R@1 by 4.1 percentage points to 0.490 and MRR by 1.3 points to 0.609, but lowers R@5 by 4.0 points to 0.776.The authors attribute new failures to centroid imbalance, as richer LLM-generated server summaries draw Stage 1 rankings away from 15 bootstrap-only servers.Source: §5.4 · p. 12, Table 7 · p. 12

5.5 Gateway Overhead

  • Gateway latency overhead versus direct MCP stdio calls was 0.8% (5 ms) over 10 trials using a lightweight health-check tool, with both paths sharing the same upstream server.For a 30-second container build, the percentage overhead is < 0.02%.Source: §5.5 · p. 13

6 Discussion

Cartograph is implemented and evaluated on a 22-server, 374-tool corpus, with discussion focused on card-generation regimes, retrieval interpretation, and system limits. The authors identify uniform LLM card generation and broader evaluation as open work.

6.1 What Is Built and Measured

  • The evaluation measures Cartograph end to end on a 22-server, 374-tool security-research corpus, including its proxy, signed cards, retrieval, Rift, and Gateway.The implementation includes a configured-LLM or deterministic-bootstrap card generator and a one-shot stdio Gateway.Source: §6.1 · p. 13

6.2 Bootstrap vs. LLM Cards

  • In the hybrid configuration, partial LLM adoption improves R@1 by 4.1 pp but degrades R@5 by 4.0 pp because LLM-described servers dominate Stage 1 ranking over bootstrap-described servers.The full retrieval benefit requires uniform LLM card generation across the entire catalog.Source: §6.2 · p. 14

6.3 Retrieval Metric Interpretation

  • R@5 measures whether the target tool reaches the agent’s shortlist, not whether the agent makes the final tool selection.Final selection also involves schema inspection, example matching, and task-specific judgment beyond embedding similarity.Source: §6.3 · p. 14

6.4 Limitations and Future Work

  • Only 7 of 22 servers have LLM-generated cards; the authors propose regenerating all 22 and rerunning the benchmark on a uniform index.They expect uniform LLM cards to outperform bootstrap and hybrid configurations, based on their centroid-imbalance analysis.Source: §6.4 · p. 14
  • Retrieval experiments use all-MiniLM-L6-v2; stronger embedding models remain unevaluated, though their marginal cost would be at index time.The authors propose evaluating stronger models to characterize the relationship between embedding quality and retrieval quality.Source: §6.4 · p. 14
  • The 49-query benchmark is author-constructed and moderate in size; independently sourced queries and n ≥ 200 are proposed to improve generalizability and R@1 statistical power.Source: §6.4 · p. 14
  • Connection pooling is proposed to reduce per-call latency from ≈650ms to ≈50ms by eliminating subprocess spawn and MCP handshake.Source: §6.4 · p. 14
  • The current trust model assumes one operator and stdio transport; cross-organization federation needs PKI, while HTTP and SSE adapters remain unimplemented.Source: §6.4 · p. 15

7 Conclusion

Cartograph replaces full-catalog disclosure with progressive disclosure through signed operator-attested cards, two-stage retrieval, and Rift. In the reported deployment, it reduces token use and improves observed retrieval over keyword overlap, while also detecting confusable clusters and adding modest Gateway overhead.

7 Conclusion

  • Cartograph changes agent-visible discovery from O(n) disclosure to O(k) progressive disclosure using operator-attested signed cards, two-stage retrieval, and Rift.In the evaluated corpus, the proxy exposes 3 tools instead of 374 detailed definitions.Source: §7 · p. 15
  • 98.8% fewer input tokens (38,250 →475 per task cycle) and observed R@5 = 0.816 versus 0.592 for Jaccard keyword retrieval on the 49-query benchmark.Source: §7 · p. 15
  • Rift detects 49 confusable clusters on the full corpus, with margin analysis supplying risk labels; Gateway overhead is 0.8% in the ten-trial experiment.Source: §7 · p. 15
  • The available LLM-generated cards remove the observed degenerate cluster across 119 tools on 7 servers, although partial adoption reduces R@5 in the hybrid index.Source: §7 · p. 15

Related papers

Loading 2609.30293v1…