PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Semantic Tool Discovery for Enterprise MCP Gateway: Architecture and Implementation

Olympia Saha, Amy Wang, Srinivasan Manoharan

arXiv:2608.23992v1cs.IRcs.AI

TL;DR

Enterprise MCP gateways must select relevant tools from large catalogs without filling the model context with tool schemas. SCOUT exposes standard MCP discovery and execution tools, using hybrid retrieval to surface relevant tools. In production, it reduces per-session tool-schema context by 99%, from 140,200 tokens to 1,300 tokens.

  • Problem

    The research problem is selecting a query-relevant subset from large MCP tool catalogs while fitting tool schemas within LLM context budgets.

    Source: §4.1 · p. 4, §4.2 · p. 5

  • Method

    SCOUT exposes MCP discovery and execution meta-tools and combines BM25 and dense retrieval with Reciprocal Rank Fusion to surface relevant tools.

    Source: §6.4 · p. 10, §9 · p. 17

  • Results

    99% reduction: SCOUT lowers per-session tool-schema context from 140,200 tokens (70.1% of a 200,000-token window) to 1,300 tokens (0.8%).

    Source: §7.2 · p. 14

  • Takeaways & Limitations

    The standard MCP meta-tool interface supports adoption by MCP-compatible clients without client-side modifications.

    Source: §9 · p. 17

    Retrieval quality depends on MCP developers’ tool descriptions, and SCOUT has no mechanism to detect or compensate for poor descriptions.

    Source: §8.2 · p. 17

Abstract

from arXiv · show

Large language model (LLM) agents invoke external tools to retrieve and reason over information beyond pretrained knowledge. The Model Context Protocol (MCP) standardizes how such tools are surfaced, and a proxy MCP server aggregates many backend servers behind a single endpoint providing a secure, governable chokepoint for authentication, policy enforcement, and observability. This architecture creates two compounding challenges: a context-engineering bottleneck where full tool schemas saturate the model context window before any user query, and a tool discoverability barrier where users and agents cannot identify the best tool among 2,000+ indexed tools across 200+ MCP servers. Prompt caching reduces reprocessing cost but neither frees context capacity nor improves accuracy. We present SCOUT (Selective Context Optimization for Universal Tooling), which reframes tool exposure as a context-selection problem, injecting only tools relevant to the current step. SCOUT surfaces two MCP meta-tools -- tool_search and execute_tool -- where tool_search performs hybrid retrieval, fusing BM25 sparse matching with dense vector search via Reciprocal Rank Fusion to return the top-k relevant tools. Backed by zero-downtime catalog update pipelines, SCOUT resolves both context saturation and tool discovery challenges. In production at PayPal, SCOUT reduces MCP tool-token consumption from 140.2k tokens (70.1% of context) to 1.3k tokens (0.8%), a 99% reduction, cutting per-query inference cost at enterprise scale. Because SCOUT is surfaced as standard MCP tools, it is model-agnostic and requires no client-side modifications.

1 Introduction

Enterprise MCP catalogs create both context saturation and tool-discovery problems. SCOUT addresses them by retrieving relevant tool schemas on demand through standard MCP meta-tools, rather than exposing the full catalog.

1 Introduction

  • Prompt caching reduces schema reprocessing cost but neither frees context capacity nor improves selection accuracy, motivating selective loading of relevant tools.At PayPal, before SCOUT, tool schemas consumed 140.2k tokens and 70% of the context window before the first user message.Source: §1 · p. 2
  • SCOUT exposes tool_search and execute_tool so agents can discover relevant tools with natural-language queries and route invocations without loading all 2,000+ schemas.It is implemented as standard MCP tools, requires no client-side modifications, and was validated across six clients spanning three major providers.Source: §1 · pp. 2–3
  • SCOUT reduces PayPal MCP tool-token consumption from 140.2k tokens (70.1% of context) to 1.3k tokens (0.8%), a 99% reduction.The reduction directly cuts per-query inference cost at enterprise scale.Source: §1 · p. 2
  • Its hybrid retrieval engine combines BM25 sparse matching and dense vector search using Reciprocal Rank Fusion, optimized for structured tool documents.Natural-language discovery targets catalogs spanning more than 200 MCP servers and 2,000+ tools.Source: §1 · p. 2
  • SCOUT enforces user-scoped authorization at retrieval and supports zero-downtime index updates with graceful degradation to full tool lists during infrastructure failures.Its insert-before-delete upsert strategy supports index management; environment-aware dual-connector routing lets users direct execution to production or development through natural-language intent.Source: §1 · pp. 2–3

2 Related Work

Prior work establishes tool-augmented reasoning and retrieval methods but leaves enterprise-scale discovery across evolving catalogs unresolved. SCOUT adapts hybrid retrieval to structured tool schemas and integrates discovery as an MCP meta-tool.

2.1 Tool-Augmented Language Models

  • Toolformer [2], ReAct [3], AutoGPT, and ReWOO [5] assume or load fixed toolsets, while ToolLLM [10] finds selection accuracy degrades with catalog size and Gorilla [11] requires retraining for new tools.SCOUT instead operates at the retrieval layer, leaving the LLM unchanged and adapting dynamically to evolving tool catalogs.Source: §2.1 · p. 3

2.2 Retrieval-Augmented Generation

  • DPR [14], BEIR [15], hybrid retrieval [12], and Reciprocal Rank Fusion motivate combining dense and sparse search, which SCOUT adapts to parameter-rich executable schemas.RAG [6] demonstrated inference-time dense retrieval without retraining; Lumer and Subbiah [13] report hybrid retrieval outperforming either modality alone in agentic retrieval settings.Source: §2.2 · p. 3

2.3 Vector Databases

  • Milvus supports native BM25 sparse vectors and dense HNSW indexing, with in-database RRFRanker fusion and self-hosting suited to enterprise data-residency requirements.This enables hybrid search through one query interface without external preprocessing or ranked-list merging.Source: §2.3 · p. 3

2.4 Retrieval in Agentic Systems

  • Lumer and Subbiah [13] find retrieval effectiveness varies significantly across agent harnesses, motivating SCOUT's first-class MCP meta-tool for retrieving structured executable tool schemas.Their study evaluates lexical and semantic retrieval across Claude Code, OpenAI Codex CLI, and Gemini CLI on document corpora; SCOUT targets parameter-rich, domain-specific tool schemas instead.Source: §2.4 · p. 4

3 Background: The AI Proxy Platform

PayPal’s AI Proxy provides a governed interface to internal MCP servers, combining tool aggregation, access control, and audit logging. Its platform model unifies upstream services behind proxy endpoints, while layered caching supports tool discovery.

3 Background: The AI Proxy Platform

  • PayPal’s FastAPI-based AI Proxy aggregates internal MCP tools, enforces per-user and per-proxy authorization, and provides structured audit logging.It acts as a single governed interface to a growing ecosystem of internal MCP servers.Source: §3 · p. 4

3.1 Core Data Model

  • The platform maps upstream MCP Servers into Proxy Servers with unified tool namespaces, and optional user mappings restrict access to selected servers.At PayPal, it spans 100+ internal MCP servers and 2,000+ tools across analytics, engineering, customer operations, and developer productivity.Source: §3.1 · p. 4

3.2 Tool Caching

  • A three-tier cache—MemoryCache, Redis, and a MySQL tool_cache table with a 24-hour TTL—provides the ground-truth registry for semantic indexing.Live MCP connections handle execution, while cached schemas support discovery independently of upstream response times.Source: §3.2 · p. 4

4 Problem Formulation

SCOUT frames tool discovery as selecting a small, relevant, authorized subset of a large catalog within context and latency constraints. The formulation addresses why full-catalog injection fails and specifies retrieval, governance, compatibility, freshness, and fallback requirements.

4.1 Formal Problem Setup

  • SCOUT selects k ≪ N relevant tools for a natural-language query while respecting user authorization, LLM context budget, and low end-to-end latency.In the PayPal deployment, the corpus contains 2,000+ tools across 100+ MCP servers; each schema encodes a tool’s name, description, and parameters.Source: §4.1 · pp. 4–5
  • Tool names are server-qualified to resolve collisions, with longest-prefix matching; callers can tune k from 1 to 20, defaulting to 5.The enhanced name combines a sanitized server name with the tool name, and the configurable k controls the recall–context-budget tradeoff without redeployment.Source: §4.1 · p. 5

4.2 Why Full Tool Injection Fails

  • 140,200 schema tokens consume 70.1% of a 200,000-token context window before a query, leaving less room for history, instructions, and reasoning.This is the PayPal deployment’s aggregate cost of injecting all tool schemas at conversation start.Source: §4.2 · p. 5
  • Irrelevant schemas can impair selection: Liu et al. [9] describe reasoning degradation when relevant information is buried, while 1,721+ irrelevant tools can surround the relevant schema.At enterprise catalog scale, manual inspection of 2,000+ tools across more than 200 MCP servers is impractical, and base MCP provides no directory or search interface.Source: §4.2 · p. 5
  • Prompt caching amortizes repeated encoding costs but does not reduce context consumption, mitigate accuracy degradation, or solve tool discoverability.The model still attends to the full catalog on every forward pass.Source: §4.2 · p. 5

4.3 Design Requirements

  • SCOUT’s requirements include high-probability top-k recall, reducing tool-token use from O(N) to O(k), search latency below 500 ms (p95), and retrieval-layer authorization.The retrieved tools must respect each user’s server-access boundaries rather than relying on post-processing filters.Source: §4.3 · p. 5
  • SCOUT also requires zero-downtime catalog updates, standard-MCP client agnosticism, and graceful degradation; Gorilla-style fine-tuning [11] conflicts with freshness and client agnosticism.If embeddings fail, SCOUT falls back to BM25-only search in Milvus; if the index is absent, it injects all available tools, preserving MCP compatibility but temporarily losing context efficiency.Source: §4.3 · p. 5

5 System Architecture

SCOUT’s architecture combines schema ingestion, a hybrid Milvus index, search and execution meta-tools, and lifecycle management to expose relevant MCP tools while keeping the catalog synchronized. Its operation spans selective tool discovery, retrieval, execution, and environment-aware routing.

5.1 Two-Phase Query Protocol

  • 140,200 tool-definition tokens are reduced to approximately 1,300 through a two-phase protocol that searches for relevant schemas, then routes execution across 2,000+ tools.Clients receive only tool_search and execute_tool; tool descriptions specify the required order and prohibit guessing tool names. The standard MCP description-driven workflow works across validated clients without special fine-tuning, protocol extensions, or client modifications.Source: §5.1 · pp. 6–7

5.2 Conditional Meta-Tool Injection

  • SCOUT injects meta-tools only when the client or proxy is enabled and the RAG index is initialized and non-empty.Operators can enable SCOUT for client types or selected proxy IDs; otherwise clients receive the conventional full-schema tools/list response.Source: §5.2 · p. 7

5.3 Index Lifecycle Management

  • Milvus indexes tools only from ACTIVE servers, removing them on degradation, deactivation, or rejection and re-ingesting them after recovery.Scheduled catalog checks also refresh changed schemas and invalidate tool caches so the index reflects current definitions.Source: §5.3 · p. 7

5.4 Scope-Based Search Filter

  • Search defaults to all ACTIVE servers, while my_servers mode restricts results to servers the user added to their proxy using a Milvus query-time metadata filter.Applying the scope filter inside Milvus enforces the restriction during retrieval rather than after results are returned.Source: §5.4 · p. 7

5.5 Ingestion Pipeline

  • Ingestion embeds structured tool documents into 3,072-dimensional vectors and uses insert-before-delete updates to avoid search downtime.Ingestion is triggered by activation or recovery, manual retrieval, or catalog changes; it loads schemas from a three-tier cache and upserts chunks into Milvus.Source: §5.5 · p. 8

5.6 Retrieval Pipeline

  • Each tool_search call retrieves expanded dense and BM25 candidate lists, fuses them with Reciprocal Rank Fusion (k=60), and returns the final top-k schemas.Dense search uses text-embedding-3-large and cosine similarity; each branch fetches top-k×3 candidates. If the embedding call fails, retrieval falls back to BM25-only search.Source: §5.6 · p. 8

5.7 Dual Connector Support and Environment-Aware Routing

  • Separate production and development connectors expose independent meta-tool pairs, and natural-language intent selects the environment and matching tool without manual toggling.For example, asking to query production BigQuery lets the LLM choose the production connector and resolve the specific tool in one search.Source: §5.7 · pp. 8–9

6 Implementation Details · 6.1 Tool Document Construction · 6.2 Milvus Collection Schema

SCOUT constructs structured tool documents with semantic and server-context anchors, then stores them in a Milvus schema supporting dense and BM25 sparse retrieval.

6.1 Tool Document Construction

  • Tool documents combine tool names, server categories and descriptions, tool descriptions, and human-readable parameter details.The parameter rendering includes types, enum values where present, and required/optional status.Source: §6.1 · p. 9
  • Including server descriptions supplies domain context that improves recall when queries describe an area of expertise rather than a specific tool.This also helps disambiguate tools across servers.Source: §6.1 · p. 9
  • The document template exposes tool, category, server-description, description, and parameter fields as semantic anchors for matching natural-language queries.Parameters are rendered with their type, required/optional status, and description.Source: §6.1 · p. 9
  • Deterministic {server_name}_{tool_name} chunk IDs enable idempotent upserts, while entity_source stores {server_name}.tool for Milvus retrieval filtering.The entity_source value is used as the Milvus filter expression.Source: §6.1 · p. 9

6.2 Milvus Collection Schema

  • The Milvus collection stores seven fields per chunk, including text with BM25 analysis, a 3,072-dimensional dense embedding, an automatically generated sparse vector, and serialized chunk JSON.The other fields are chunk_id, entity_name, and entity_source.Source: §6.2 · p. 9
  • Milvus indexes the dense embedding with HNSW using COSINE and the BM25 sparse field with SPARSE_INVERTED_INDEX.Source: §6.2 · p. 9

6.3 Embedding Strategy · 6.4 Hybrid Retrieval with RRF Fusion

SCOUT embeds tools and queries with text-embedding-3-large, then combines dense and BM25 retrieval through Reciprocal Rank Fusion. Its over-fetching strategy widens the candidate pool so results favored by either modality can reach the final ranking.

6.3 Embedding Strategy

  • text-embedding-3-large generates 3,072-dimensional vectors and was chosen for its semantic capture quality.Embedding calls are batched in groups of 16, with automatic back-off retries for rate-limit, connection, and timeout errors.Source: §6.3 · p. 9

6.4 Hybrid Retrieval with RRF Fusion

  • At query time, dense search embeds the query and retrieves top_k × 3 candidates from an HNSW index using cosine similarity, while BM25 searches raw query text for the same candidate count.The BM25 search uses the SPARSE_INVERTED_INDEX.Source: §6.4 · p. 9
  • Milvus’s RRFRanker with k=60 fuses the two ranked lists into a single list of top_k results.The dense and sparse result sets are fused using Reciprocal Rank Fusion.Source: §6.4 · p. 9
  • With top_k = 5, each branch retrieves 15 candidates, giving RRF a 30-candidate pool that can include results ranked highly by only one modality.This over-fetching prevents a document ranked 10th by dense search but 2nd by BM25 from being excluded before fusion; it adds modest retrieval work in exchange for better quality when rankings disagree.Source: §6.4 · p. 10
  • BM25 matches exact tool names and parameter keywords, while dense retrieval handles paraphrases and intent-level queries.The passage contrasts the exact name “create_pull_request” with the intent-level query “open a new GitHub PR.”Source: §6.4 · p. 10
  • If the OpenAI embedding call fails during an API outage, the system falls back to BM25-only search to preserve partial functionality.Source: §6.4 · p. 10

6.5 Meta-Tool Interface Specification · 6.6 Observability · Session Entry Volume

SCOUT’s sections specify how its search and execution meta-tools constrain tool discovery and routing, and how structured events support observability from session entry through execution.

6.5 Meta-Tool Interface Specification

  • tool_search uses a task-description query, optional top_k and scope parameters, and returns ranked tool descriptors with full input schemas.top_k defaults to 5 and is clamped to [1, 20]; scope can restrict retrieval to personally mapped or all active servers.Source: §6.5 · p. 10
  • The tool_search response is an MCP text-content block containing the echoed query and ordered matches for LLM review before a tool call.Each match includes the enhanced tool name, server name, description, and inputSchema.Source: §6.5 · p. 10
  • In dual-connector deployments, environment identifiers in each connector’s tool_search description are the sole signal for routing between environments.The labels require no schema changes or additional parameters.Source: §6.5 · p. 10
  • execute_tool requires the exact tool_name and server_name from search results, plus arguments conforming to the returned inputSchema.Both name parameters warn “Do not guess”; the proxy routes using server_name and passes the upstream response through unmodified.Source: §6.5 · p. 10

6.6 Observability

  • SCOUT emits structured log events at every stage of its two-phase query protocol, supporting a Splunk dashboard across session entry and tool execution.The events are used to derive four monitoring tiers.Source: §6.6 · p. 10

Session Entry Volume

  • The tools_list_tool_search event records each tools/list call while SCOUT is active, enabling adoption and session-volume tracking by proxy.Per-proxy breakdowns reveal high-traffic endpoints and support capacity planning.Source: Session Entry Volume · p. 11

RAG Search Analytics · Fallback Rate Monitoring · Tool Execution Analytics

SCOUT’s analytics use search, fallback, and execution events to monitor result quality, discoverability, scope, latency, reliability, backend load, and failures.

RAG Search Analytics

  • Search result counts and zero-result rates are tracked by proxy and query scope to assess result quality.Source: RAG Search Analytics · p. 11
  • Matched-server frequency and my_servers versus all_servers invocation ratios reveal discoverable backend capabilities and user intent patterns.Source: RAG Search Analytics · p. 11
  • Search latency is monitored at p50, p95, and p99 over time to detect performance regressions as the index scales.Source: RAG Search Analytics · p. 11

Fallback Rate Monitoring

  • Fallback rate is the rolling-window ratio of tool_search_fallback to tool_search_completed events, measuring RAG reliability when Milvus or OpenAI embeddings are unavailable.Source: Fallback Rate Monitoring · p. 11
  • Comparing RAG-path and fallback-path latency quantifies degraded-operation performance costs and can motivate remediation of embedding-service disruptions.Source: Fallback Rate Monitoring · p. 11

Tool Execution Analytics

  • Execution events support most-invoked tool rankings and per-server call-volume tracking to guide index prioritization and monitor backend load distribution.Source: Tool Execution Analytics · p. 11
  • Failure heatmaps by tool-server pair and error class or category distinguish transient network failures from systematic tool-level defects.Source: Tool Execution Analytics · p. 11

7 Evaluation

On a 49-query benchmark, SCOUT retrieves relevant tools across a 2,000-tool production catalog while sharply reducing schema context. The evaluation also identifies ranking errors within domains and multi-step workflows as distinct failure modes.

7.1 Retrieval Quality

  • For a Slack-message query, dense retrieval finds the available Teams tool teams_send_message where BM25 returns no matches, making this a retrieval success rather than a catalog gap.The capability exists under a different product name, and hybrid retrieval bridges the vocabulary gap by matching semantic intent.Source: §7.1 · p. 12
  • Across 45 evaluable queries, SCOUT achieves Hit@1 of 84.8% (38/45), Hit@5 of 95.6% (43/45), and MRR of 0.821.On the 35-query expansion set, Hit@1 is 75.6%, while correct tools appear in the top 5 in 93.9% of evaluable cases; four catalog gaps have no corresponding MCP server.Source: §7.1 · pp. 11–12, Table 1b · p. 13

7.2 Context Efficiency

  • SCOUT reduces per-session tool-schema context from 140,200 tokens (70.1% of a 200,000-token window) to 1,300 tokens (0.8%), a 99% reduction.In the spot-check, each query retrieved approximately 2,500 tokens of schema on average with top-5 results.Source: Table 2 · p. 14, §7.2 · p. 14

7.3 Failure Mode Analysis

  • The two spot-check failure modes are intra-domain intent ranking and confusion between a workflow entry point and an intermediate step.For the calendar query, create_event ranked second behind respond_to_meeting; for the pull-request query, approve_tests ranked above generate_pr. The paper proposes intent pre-filtering and workflow-aware chunking as directions to address these errors.Source: §7.3 · pp. 14–15

7.4 Production Reliability

  • Production operational metrics were drawn from 24 hours of PayPal MCP Hub logs, and SCOUT routes queries across the full enterprise tool catalog.The paper presents these real-user activity logs as evidence of reliability and enterprise-scale adoption.Source: §7.4 · pp. 15–16

8 Discussion

SCOUT’s two-tool MCP interface enables semantic tool discovery across validated clients without client-side modifications. Its discussion identifies retrieval, feedback, synchronization, and top-k tuning limitations, alongside proposed improvements.

8.1 The Meta-Tool Protocol Pattern

  • The MCP-exposed tool_search → execute_tool pattern enabled semantic discovery without client changes across six validated MCP clients.The clients were Claude Code, Claude Desktop, ChatGPT Web and Desktop, GitHub Copilot (VS Code), Cursor, and OpenAI Codex CLI; tool descriptions guide the workflow and load from startup templates that can be updated without code changes.Source: §8.1 · p. 17

8.2 Limitations

  • Tool-description quality can cause under-retrieval; absent feedback learning, 15-minute sync intervals, and manually tuned top-k further limit retrieval.Poor descriptions may be under-retrieved regardless of modality; successful invocations do not refine embeddings or scores, schema changes may take 15 minutes to propagate absent manual force-refresh, and fixed top-k can under-retrieve broad queries or waste context on narrow ones.Source: §8.2 · p. 17

8.3 Future Directions

  • Proposed directions include success-weighted re-ranking, multi-vector tool representations, and automatic top-k calibration based on query complexity and result confidence.Feedback-weighted re-ranking could improve precision without embedding-model retraining, while multi-vector representations may disambiguate tools with broad or overlapping parameter spaces.Source: §8.3 · p. 17

9 Conclusion

SCOUT is a production semantic tool discovery system integrated into PayPal’s AI Proxy platform, combining hybrid retrieval and operational safeguards to address static provisioning limits. Its standard MCP meta-tools support adoption by compatible clients without client-side changes.

9 Conclusion

  • SCOUT is a production semantic tool discovery system for PayPal’s AI Proxy platform, addressing scalability limits of static tool provisioning through five complementary mechanisms.These are hybrid BM25 and dense retrieval with RRF fusion, 3,072-dimensional embeddings, scope-based authorization filtering, zero-downtime index management, and graceful degradation to full tool lists during infrastructure failure.Source: §9 · p. 17
  • Hybrid BM25 and dense retrieval with RRF fusion, 3,072-dimensional embeddings, and scope-based authorization filtering underpin SCOUT’s discovery approach.The conclusion also identifies zero-downtime index management and graceful degradation to full tool lists under infrastructure failure as complementary mechanisms.Source: §9 · p. 17
  • The meta-tool protocol exposes discovery and execution as standard MCP tools, enabling MCP-compatible clients to adopt SCOUT without client-side modifications.This embeds semantic tool discovery as a first-class MCP capability rather than an external pre-processing layer, and is composable with future MCP protocol extensions.Source: §9 · p. 17

Related papers

Loading 2608.23992v1…