Source-linked AI summary
RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation
Tiantian Gan, Qiyao Sun
TL;DR
Large MCP toolsets make it difficult to fit tool descriptions into prompts and distinguish among similar tools. RAG-MCP retrieves relevant tool schemas from an external index before invoking the LLM; it cuts token usage by over half and more than triples selection success under heavy load.
Problem
As MCP toolsets grow, including all descriptions can saturate the context and make similar tools harder to distinguish, complicating tool selection.
Source: §1.1 · pp. 2–3Method
RAG-MCP semantically retrieves relevant MCP candidates from an external index and provides the selected tool description to the LLM.
Source: §1.1 · p. 3, §3.2 · p. 6Results
RAG-MCP cuts token usage by over half and more than triples the success rate of naïve and keyword-based methods under heavy load.
Source: §6 · p. 11Takeaways & Limitations
RAG-MCP is presented as an on-demand toolkit approach for using large libraries of external tools with LLM agents.
Source: §6 · p. 11Retrieval precision challenges arise as MCP counts grow, motivating future hierarchical or adaptive retrieval mechanisms.
Source: Stress Test Analysis · p. 10
Abstract
from arXiv · showhide
Large language models (LLMs) struggle to effectively utilize a growing number of external tools, such as those defined by the Model Context Protocol (MCP)\cite{IntroducingMCP}, due to prompt bloat and selection complexity. We introduce RAG-MCP, a Retrieval-Augmented Generation framework that overcomes this challenge by offloading tool discovery. RAG-MCP uses semantic retrieval to identify the most relevant MCP(s) for a given query from an external index before engaging the LLM. Only the selected tool descriptions are passed to the model, drastically reducing prompt size and simplifying decision-making. Experiments, including an MCP stress test, demonstrate RAG-MCP significantly cuts prompt tokens (e.g., by over 50%) and more than triples tool selection accuracy (43.13% vs 13.62% baseline) on benchmark tasks. RAG-MCP enables scalable and accurate tool integration for LLMs.
1 Introduction
External tools extend LLM capabilities, but growing toolsets can saturate prompts and make tool selection harder. The introduction presents RAG-MCP as retrieval-based tool discovery and reports contributions in scalable retrieval and tool-use performance.
1.1 Background and Motivation
- External tools and function-calling interfaces let LLMs fetch information and perform operations beyond their built-in repertoire.Prior work spans prompting, model fine-tuning, and retrieved API documentation [3].Source: §1.1 · pp. 1–2
- Growing tool collections can exhaust prompt context and blur similar functions, while increasing the chance of choosing a suboptimal tool or hallucinating an API.The introduction also notes a reported case in which Claude chose the wrong library for a request.Source: §1.1 · pp. 2–3
- RAG-MCP stores tool descriptions in an external semantic index and retrieves query-relevant tools so the model receives only a focused subset.The index can be updated with new or changed tools without retraining the LLM.Source: §1.1 · pp. 3–4
1.2 Contributions
- The paper contributes a semantic retrieval module for MCP schemas that narrows tool choices, reduces prompt size, and supports adding tools without fine-tuning.It also reports experiments in which retrieval largely restores tool-use performance as the available function set grows.Source: §1.2 · p. 4
- RAG-MCP is presented as a practical approach to retrieval-based context management for LLM agents using extensive toolkits.The introduction frames the goal as using information about selected tools rather than providing every available tool to the model.Source: §1.2 · p. 4
2 Related Work
Related work covers tool-use methods, retrieval-augmented generation, and the Model Context Protocol. Together, these strands frame MCP discovery as a retrieval problem in an expanding tool ecosystem.
2.1 Tool Use in LLMs
- Tool-use research includes API-calling methods, reasoning interleaved with actions, browser-based training, and production plugins.The cited examples address tool use for tasks including search, calculators, external environments, and long-form question answering.Source: §2.1 · p. 5
2.2 Retrieval-Augmented Generation
- RAG retrieves relevant passages from a dense vector index at inference time, and later variants adapt retrieval per token or query [4].The paper treats MCP discovery as a retrieval subproblem separate from core text generation.Source: §2.2 · p. 5
2.3 Model Context Protocol
- MCP packages prompts, authentication, and parameter schemas into modular servers, and repositories had grown to 4,400+ servers as of April 2025.The passage identifies scalable discovery and validation as needs associated with this growth.Source: §2.3 · p. 5
3 Methodology
The methodology frames tool discovery as retrieval to address prompt bloat as MCP pools grow. RAG-MCP retrieves and optionally validates candidates, then gives the selected schema to the LLM for execution.
3 Methodology
- RAG-MCP is framed as a retrieval-augmented framework that dynamically retrieves relevant MCPs to mitigate performance degradation as server counts grow.The approach separates tool discovery from generation so the model need not receive the full registry.Source: §3 · p. 5, §3.3 · p. 7
3.1 Prompt Bloat and the MCP Stress Test
- The MCP stress test varies pool size and measures selection accuracy, task success, prompt-token usage, and latency with one relevant schema among distractors.Its design draws a parallel to NIAH, where performance drops as the context grows.Source: §3.1 · p. 6
3.2 RAG-MCP Framework
- A semantic retriever searches an external MCP index for top-k candidates; optional synthetic-example checks test compatibility before invocation.Only the best MCP description and its tool-use parameters are passed to the LLM for planning and execution.Source: §3.2 · p. 6
3.3 Three-Step Pipeline Diagram
- The pipeline encodes the task, ranks and optionally tests MCP schemas, then gives the selected schema and parameters to the LLM for function-call execution.Separating discovery from generation is presented as allowing scaling to hundreds or thousands of MCPs without prompt bloat or decision fatigue.Source: §3.3 · p. 7, Figure 2 · p. 7
3.4 Discussion
- The discussion says dynamic toolset narrowing reduces prompt tokens and decision complexity, restoring and sometimes improving task success rates.An external index supports adding MCPs without retraining, while on-demand server activation avoids practical limits on simultaneous instantiation.Source: §3.4 · pp. 7–8
4 Experiments
The experiments compare MCP selection strategies and assess performance as the candidate pool grows. RAG-MCP has the highest reported accuracy and lower prompt-token use than Blank Conditioning.
4.1 Stress Test
- The stress test varied candidate MCP counts from 1 to 11100, using one ground-truth server and distractors for each of 20 web-search tasks.The reported trend is non-monotonic, with retrieval precision and throughput degrading as the registry scales to thousands of MCPs.Source: §4.1 · p. 8, Figure 3 · p. 8, Results · p. 9
Setup
- Methods were evaluated on the web-search subset of MCPBench, with 20 trials per baseline and up to 10 interaction rounds per trial.Qwen-max-0125 was the base LLM, and Deepseek-v3 was used to evaluate answer correctness.Source: Setup · p. 9
Baselines
- The comparison includes Blank Conditioning with all MCP descriptions, keyword-based Actual Match, and RAG-MCP semantic retrieval of the top candidate.The methods differ in how they reduce or present the candidate pool to the model.Source: Baselines · p. 9
Metrics
- The evaluation tracks selection accuracy, average prompt tokens, and average completion tokens.Final-answer judgment is automated using a Llama-based verifier.Source: Metrics · p. 9
Results
- RAG-MCP reached 43.13% accuracy, versus 18.20% for Actual Match and 13.62% for Blank Conditioning, while averaging 1084 prompt tokens versus Blank Conditioning’s 2133.84.Its average completion tokens were 78.14, compared with 23.60 for Actual Match; the authors describe this trade-off as beneficial alongside higher accuracy and task success.Source: Table 1 · p. 10, Metrics · p. 10
5 Analysis
The stress test shows high performance in small-to-moderate MCP pools but retrieval-precision challenges as registries grow. The analysis attributes stronger selection to focused schemas and reduced prompt overhead.
Stress Test Analysis
- With minimal candidate pools, MCP positions below 30 exhibit success rates above 90%; occasional yellow patches at higher positions suggest certain MCPs remain well-aligned to specific queries, providing robustness in extensive pools.Source: Stress Test Analysis · p. 10
- Beyond position ~100, purple dominates and retrieval precision diminishes, while the analysis says precision challenges arise as the MCP pool grows.The paper describes high accuracy in small-to-moderate pools but notes that retrieval precision challenges emerge as the total number of MCPs increases.Source: Stress Test Analysis · p. 10
5.1 Analysis of RAG-MCP Results
- Injecting only the single most relevant schema avoids distraction from irrelevant descriptions and leaves more context for reasoning about the task.The analysis also says focused filtering results in clearer decision boundaries.Source: §5.1 · p. 11
- RAG-MCP uses more completion tokens than Actual Match, an overhead the authors associate with more thorough reasoning and verification that correlates with higher accuracy.The reported explanation is a correlation, not a demonstrated causal effect.Source: §5.1 · p. 11
- The authors conclude that retrieval-augmented MCP selection curbs prompt bloat and enhances tool-selection reliability for scalable external-tool integration.They characterize RAG-MCP as a compelling solution for scalable external tool integration.Source: §5.1 · p. 11
6 Conclusion
RAG-MCP retrieves only the most relevant schema for each query, reducing prompt size and increasing selection accuracy. The conclusion also identifies on-the-fly indexing without retraining and several directions for future work.
6 Conclusion
- Prompt token usage falls by over half compared with feeding all tools at once.The conclusion presents prompt-size reduction as a result of focused retrieval.Source: §6 · p. 11
- Selection accuracy more than triples the success rate of naïve and keyword-based methods under heavy load.Source: §6 · p. 11
- New MCPs can be indexed on the fly without retraining the model.The conclusion frames this as preserving extensibility.Source: §6 · p. 11
- Future work will refine retrieval at extreme scale and explore multi-tool workflows and real-world agent deployments.Source: §6 · p. 11