PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Semantic Tool Discovery for Enterprise MCP Gateway: Architecture and Implementation

Olympia Saha, Amy Wang, Srinivasan Manoharan

arXiv:2608.23992v1cs.IRcs.AI

TL;DR

Enterprise MCP deployments make full tool-schema injection expensive and tool selection difficult across large catalogs. SCOUT treats exposure as context selection, using hybrid BM25–dense retrieval with RRF through standard MCP meta-tools. At PayPal, it reduced tool-schema context from 140.2k tokens to 1.3k tokens, a 99% reduction, while achieving 95.6% Hit@5 on the full benchmark.

  • Problem

    Enterprise deployments face context saturation, degraded tool selection, and impractical discovery when exposing 2,000+ tools across more than 200 MCP servers.

  • Method

    SCOUT exposes standard MCP tool_search and execute_tool meta-tools, selecting authorized relevant tools through hybrid BM25 and dense retrieval fused with Reciprocal Rank Fusion.

  • Results

    SCOUT achieved 95.6% Hit@5 and reduced per-session tool-schema context from 140,200 tokens to 1,300 tokens, a 99% reduction.

  • Takeaways & Limitations

    Standard MCP discovery and execution tools enable adoption across compatible clients without client-side modifications while embedding semantic discovery within the MCP ecosystem.

  • Takeaways & Limitations

    Tool descriptions vary in quality, feedback-loop learning is unsupported, schema updates may take up to 15 minutes to propagate, and top-k requires manual tuning.

Abstract

from arXiv · show

Large language model (LLM) agents invoke external tools to retrieve and reason over information beyond pretrained knowledge. The Model Context Protocol (MCP) standardizes how such tools are surfaced, and a proxy MCP server aggregates many backend servers behind a single endpoint providing a secure, governable chokepoint for authentication, policy enforcement, and observability. This architecture creates two compounding challenges: a context-engineering bottleneck where full tool schemas saturate the model context window before any user query, and a tool discoverability barrier where users and agents cannot identify the best tool among 2,000+ indexed tools across 200+ MCP servers. Prompt caching reduces reprocessing cost but neither frees context capacity nor improves accuracy. We present SCOUT (Selective Context Optimization for Universal Tooling), which reframes tool exposure as a context-selection problem, injecting only tools relevant to the current step. SCOUT surfaces two MCP meta-tools -- tool_search and execute_tool -- where tool_search performs hybrid retrieval, fusing BM25 sparse matching with dense vector search via Reciprocal Rank Fusion to return the top-k relevant tools. Backed by zero-downtime catalog update pipelines, SCOUT resolves both context saturation and tool discovery challenges. In production at PayPal, SCOUT reduces MCP tool-token consumption from 140.2k tokens (70.1% of context) to 1.3k tokens (0.8%), a 99% reduction, cutting per-query inference cost at enterprise scale. Because SCOUT is surfaced as standard MCP tools, it is model-agnostic and requires no client-side modifications.

1 Introduction

Enterprise LLM agents need external tools across heterogeneous repositories, while MCP standardizes tool access and proxy servers centralize aggregation and governance. As enterprise catalogs expand, selecting the right capability among thousands of tools becomes a critical infrastructure challenge.

  • MCP provides a standardized JSON-RPC interface for connecting LLM agents to external data sources and services.
  • Proxy MCP servers aggregate multiple backend servers behind one endpoint while centrally managing authentication, policy enforcement, and observability.
  • Enterprise tool catalogs create a discovery problem for agents selecting among thousands of schemas and users navigating more than 200 MCP servers.

2 Related Work

Prior tool-use and retrieval research established agentic tool calling, semantic retrieval, and hybrid lexical–dense search, but did not address enterprise-scale discovery across thousands of MCP tools. The paper builds on these foundations and related agentic retrieval findings.

  • Toolformer and ReAct established tool calling and interleaved reasoning–action, while AutoGPT and ReWOO extended orchestration but still assumed all tools fit in context.
  • Existing approaches did not address scalable discovery across hundreds or thousands of enterprise tools, the gap targeted by this work.
  • RAG and DPR demonstrated meaning-based dense retrieval, while BEIR showed BM25 remains competitive, motivating a hybrid retrieval design.
  • Milvus supports BM25 sparse vectors, dense HNSW indexing, and in-database RRF fusion, providing a self-hostable implementation for enterprise search.
  • Agentic retrieval differs from one-shot standalone search because agents iteratively choose queries, retrieval breadth, and whether results require refinement.

3 Background: The AI Proxy Platform

PayPal’s AI Proxy is governed middleware that aggregates internal MCP services behind unified proxy endpoints. Its data model, authorization mappings, and layered caching support centralized access control, auditability, and scalable tool discovery.

  • PayPal’s FastAPI-based AI Proxy provides one governed interface to internal MCP servers, combining aggregation, access control, and structured audit logging.
  • Core Data Model: An MCP Server represents an upstream service, while a Proxy Server aggregates one or more MCP Servers into a unified client-facing namespace.
  • Core Data Model: User Proxy MCP Server Mapping restricts an individual user to a subset of servers within a Proxy Server.
  • Core Data Model: The platform spans 100+ internal MCP servers and 2,000+ tools across analytics, engineering, customer operations, and developer productivity.
  • A three-tier cache uses in-memory storage, Redis, and a MySQL tool_cache table with a 24-hour TTL to maintain the tool registry for semantic indexing.
  • Live MCP connections handle execution while cached schemas support discovery, decoupling retrieval latency from upstream-server responsiveness.

4 Problem Formulation

The paper formulates tool discovery as selecting a small, authorized, relevant subset of schemas under context and latency constraints. Full injection fails through context saturation, reduced accuracy, and impractical manual discovery, motivating explicit retrieval requirements.

  • 4.1 Formal Problem Setup: The system selects S ⊆ T with |S| = k ≪ N to maximize query relevance while limiting context usage and preserving low latency.
  • 4.1 Formal Problem Setup: Each tool is represented by a schema containing its name, description, and parameter specification, with schema size measured in tokens.
  • 4.1 Formal Problem Setup: Retrieved tools must satisfy per-user server authorization constraints enforced during retrieval rather than after filtering.
  • 4.1 Formal Problem Setup: Enhanced server-prefixed names resolve non-unique tool names, using a longest-prefix-wins rule when multiple prefixes match.
  • 4.1 Formal Problem Setup: The configurable k value is clamped to [1, 20] and defaults to 5, allowing callers to trade recall against context budget.
  • 4.2 Why Full Tool Injection Fails: 140,200 tokens of schemas consume 70.1% of a 200,000-token context window before user input, causing context saturation.
  • 4.2 Why Full Tool Injection Fails: Relevant schemas can be buried among 1,721+ irrelevant tools, worsening selection accuracy through the lost-in-the-middle phenomenon.
  • 4.3 Design Requirements: SCOUT requires high recall, O(k) context consumption, sub-500 ms p95 search latency, and retrieval-time authorization enforcement.

5 System Architecture

SCOUT exposes only tool_search and execute_tool to clients, using description-driven orchestration to retrieve and route relevant tools while supporting controlled rollout, health-aware indexing, scoped search, and multi-environment dispatch.

  • Meta-tool exposure: Tool_search retrieves ranked schemas, while execute_tool routes calls to the appropriate upstream MCP server using the selected tool and server identity.The architecture separates discovery from execution through dedicated meta-tools.
  • Meta-tool exposure: SCOUT replaces the full catalog with two meta-tools, tool_search and execute_tool, so clients receive only a stub list while the full 2,000+ tool catalog stays server-side.The two-phase behavior is specified through tool descriptions and requires no fine-tuning, protocol extensions, or client-side modifications.
  • Deployment control: SCOUT rollout is independently controlled at client and proxy levels, with meta-tool injection gated by configuration and a non-empty initialized RAG index.This enables selected-client and selected-proxy deployment without affecting standard proxy behavior.
  • Catalog lifecycle: The index contains tools only from ACTIVE servers, with server-state transitions triggering ingestion, removal, or degradation behavior.Health-aware indexing keeps searchable tools aligned with backend availability.
  • Retrieval scope: Search supports all_servers and my_servers scopes, enforcing server restrictions inside Milvus through query-time metadata filters.The default searches all ACTIVE servers, while my_servers limits retrieval to servers explicitly added by the user.
  • Environment routing: Production and development connectors can expose separate meta-tool pairs, allowing natural-language intent to select the environment without manual toggling or client configuration.The same description-driven orchestration used for two-phase discovery also handles environment-aware dispatch.

6 Implementation Details

SCOUT implements searchable tool documents, hybrid dense-sparse retrieval with RRF fusion, structured meta-tool interfaces, and observability for discovery, fallback, and execution behavior.

  • Tool representation: Each tool becomes an embedding document combining its name, server category and description, tool description, and parameter details.Parameter renderings include types, enum values where present, and required or optional status; server descriptions provide broader domain context.
  • Tool representation: Deterministic server-and-tool IDs support idempotent upserts, while entity_source enables server filtering during retrieval.The identifier format is {server_name}_{tool_name}, and the source field stores the server name as {server_name}.tool.
  • Index implementation: Milvus stores dense embeddings, BM25-enabled text, sparse vectors, metadata, and serialized chunks, using HNSW and sparse inverted indexes for retrieval.Dense vectors have dimension 3,072, and the collection maintains separate dense and sparse indexes.
  • Hybrid retrieval: Hybrid retrieval combines dense semantic search and BM25 keyword search, then fuses their rankings with Milvus RRFRanker using k=60.Dense and sparse branches each over-fetch top_k × 3 candidates; BM25 favors exact names and keywords, while dense retrieval handles paraphrased intent.
  • Hybrid retrieval: Over-fetching creates a wider fusion pool so documents highly ranked by only one modality can still enter the final top_k results.With top_k = 5, each branch retrieves 15 candidates before fusion, producing a 30-candidate pool.
  • Meta-tool interfaces: Tool_search accepts a natural-language query, bounded top_k, and optional scope, returning ordered matches with enhanced names, server names, descriptions, and full input schemas.The default top_k is 5 and the allowed range is 1 to 20.
  • Observability: Structured events support monitoring of adoption, result quality, fallback reliability, backend surfacing, latency, and execution failures.Fallback events specifically track degraded operation when Milvus is unreachable or embedding calls fail.

7 Evaluation

SCOUT was evaluated on retrieval quality, context efficiency, latency, failure modes, and production usage. It generally surfaced relevant tools within the top five and sharply reduced context consumption, while revealing ranking and workflow-disambiguation limitations.

  • Retrieval Quality: 93.9% of expansion-set queries returned correct tools within the top five, while four catalog gaps lacked corresponding MCP servers.The gaps involved Confluence documentation, screenshot capture, and general-purpose web scraping.
  • Retrieval Quality: Hybrid retrieval bridged vocabulary mismatch by mapping a Slack messaging request to the available Microsoft Teams tool when BM25 alone returned no match.This was classified as a retrieval success rather than a catalog gap.
  • Retrieval Quality: Domain performance was strongest for analytics and observability, while SRE, infrastructure, and productivity achieved Hit@1 rates of 78%, 78%, and 75%, respectively.Failures included runbook coverage, OOMKill-specific ranking, and productivity retrieval cases.
  • Operational Performance: Production server-side processing averaged 572 ms, with a 440 ms median and 936 ms P95 across 1,921 requests.Measurement covered hybrid search, cached schema resolution, and response formatting; warm-cache retrieval reached a 232 ms minimum.
  • Operational Performance: The latency distribution was right-skewed, centered at 300-400 ms, with a long tail attributed to cold-cache retrievals and peak-load contention.The distribution covered 8,209 production searches over seven days.
  • Context Efficiency: 99% lower context consumption reduced per-session tool-schema overhead from 140,200 tokens to 1,300 tokens, or from 70.1% to 0.8% of the context window.Spot-check queries averaged approximately 2,500 retrieved schema tokens for top-five results.
  • Failure Modes: A 15-query spot-check identified intra-domain ranking errors and workflow-step confusion, including sub-steps ranking above workflow entry points.The authors propose workflow-aware chunking that groups related steps under a single entry-point document.

8 Discussion

SCOUT embeds semantic tool discovery into the MCP interface, enabling compatible clients to use a two-tool workflow without client changes. The paper also identifies limitations in description quality, feedback learning, index freshness, and top-k selection.

  • 8.1 The Meta-Tool Protocol Pattern: Exposing tool_search as an MCP tool gives six validated clients semantic discovery through the standard tool_search → execute_tool workflow without client-side modifications.The workflow is guided by tool descriptions and spans multiple LLM providers and interface types.
  • 8.2 Limitations: SCOUT lacks safeguards for poorly authored tool descriptions, which may cause systematic under-retrieval regardless of retrieval modality.The system currently has no mechanism to detect or compensate for description-quality variation.
  • 8.2 Limitations: The system does not use successful invocations as feedback to refine embeddings or future retrieval scores.Feedback-weighted re-ranking is identified as a future direction rather than a current capability.
  • 8.2 Limitations: Index changes may take up to 15 minutes to propagate because synchronization runs every 15 minutes, while urgent updates require manual force-refresh.This creates a freshness boundary for time-sensitive schema changes.
  • 8.2 Limitations: Manual top-k tuning may under-retrieve broad queries or waste context on narrow ones because the system lacks automatic calibration.The proposed future direction is to adjust returned-tool count based on query complexity, overlap, or confidence.

9 Conclusion

The paper presents a production semantic tool-discovery system for PayPal’s AI Proxy, combining hybrid retrieval, authorization, resilient indexing, and graceful degradation. Standard MCP meta-tools allow adoption by compatible clients without client-side modifications.

  • 9 Conclusion: SCOUT addresses static tool-provisioning scalability through hybrid BM25 and dense retrieval, 3,072-dimensional embeddings, authorization filtering, resilient index management, and graceful degradation.These mechanisms are integrated into PayPal’s production AI Proxy platform.
  • 9 Conclusion: Standard MCP tools for discovery and execution make semantic tool discovery adoptable by any MCP-compatible client without client-side modifications.The design treats discovery as a first-class MCP capability rather than an external preprocessing layer.
Loading 2608.23992v1…