Source-linked AI summary
Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions
Mohammed Mehedi Hasan, Hao Li, Gopi Krishnan Rajbahadur, Bram Adams, Ahmed E. Hassan
TL;DR
MCP agents depend on tool descriptions, yet missing parameter names and formats can make tool invocation unreliable. This paper identifies description components and smells, finding that rubric-aligned augmentation improves performance but can increase execution cost and does not help universally.
Problem
Method
The paper identifies tool-description components, scores their quality, and maps subthreshold scores to six corresponding description smells.
Source: §4.1.3 · p. 11, §4.1.4 · p. 13Results
A 5.85 percentage-point task-success gain and 15.12% Average Evaluator Score increase accompany regressions in 16.67% of cases and higher execution cost.
Source: §6.3 · p. 37, §9 · p. 43Takeaways & Limitations
Tool descriptions should be treated as configurable engineering artifacts with cost-aware, context-sensitive augmentation rather than universally expanded.
Source: §9 · p. 43The paper illustrates task generation using the get_historical_stock_prices tool in the yfinance MCP server, including three example retrieval tasks.
Source: §4.4.2 · p. 18
Abstract
from arXiv · showhide
The Model Context Protocol (MCP) introduces a standard specification that defines how Foundation Model (FM)-based agents should interact with external systems by invoking tools. However, to understand a tool's purpose and features, FMs rely on natural-language tool descriptions, making these descriptions a critical component in guiding FMs to select the optimal tool for a given (sub)task and to pass the right arguments to the tool. While defects or smells in these descriptions can misguide FM-based agents, their prevalence and consequences in the MCP ecosystem remain unclear. Hence, we examine 856 tools spread across 103 MCP servers empirically, assess their description quality, and their impact on agent performance. We identify six components of tool descriptions from the literature, develop a scoring rubric utilizing these components, and then formalize tool description smells based on this rubric. By operationalizing this rubric through an FM-based scanner, we find that 97.1% of the analyzed tool descriptions contain at least one smell, with 56% failing to state their purpose clearly. While augmenting these descriptions for all components improves task success rates by a median of 5.85 percentage points and improves partial goal completion by 15.12%, it also increases the number of execution steps by 67.46% and regresses performance in 16.67% of cases. These results indicate that achieving performance gains is not straightforward; while execution cost can act as a trade-off, execution context can also impact. Furthermore, component ablations show that compact variants of different component combinations often preserve behavioral reliability while reducing unnecessary token overhead, enabling more efficient use of the FM context window and lower execution costs.
1 Introduction
MCP tool descriptions guide foundation-model agents’ tool discovery, selection, parameterization, and orchestration, but defects can mislead agents and increase execution costs. This study systematically investigates description smells and their effects across 856 tools from 103 MCP servers.
MCP workflow
- MCP exposes tool names, descriptions, and input schemas so foundation-model agents can discover capabilities, plan solutions, infer parameters, and issue tool calls.The workflow retrieves metadata, supplies it with the user query, executes the selected call, and returns the response for final synthesis.Source: §1 · p. 2, Figure 1 · p. 2
Motivation
- Tool descriptions function as both behavioral specifications and contextual instructions, so defects can cause wrong tool choices, invalid arguments, unnecessary steps, and reduced reliability.They define intended functionality, constraints, and usage cues while shaping model reasoning and decision-making [99] [64].Source: §1 · pp. 2–3
Efficiency tension
- Augmenting descriptions may improve semantic guidance and performance, but repeatedly injected metadata consumes context capacity and increases execution cost.The introduction frames compact representations as necessary for preserving effectiveness while limiting token overhead, alongside developments such as Agent Skills and Tool Search.Source: §1 · p. 3
Research gap
- No systematic study had established how prevalent MCP tool-description smells are or whether recommended writing practices improve agent behavior under realistic workloads.Prior work addressed tool-calling challenges, code smells, and prompt components, but not MCP description quality and downstream performance [38].Source: §1 · p. 3
Study design
- The study scans 856 tools across 103 MCP servers with a structured rubric, automatically repairs identified smells, and evaluates agents using the MCP Universe benchmark [59].The empirical design targets description quality and its downstream effects on foundation-model agent performance.Source: §1 · p. 3
RQ-1: To what extent do MCP tools’ descriptions contain smells?
Across 856 MCP tool descriptions, smells are pervasive: 97.1% contain at least one, and 56% have unclear purposes. These issues affect both official and community-maintained servers, revealing widespread difficulty producing high-quality descriptions.
- The study establishes an empirical baseline by quantifying smell prevalence and distribution across official and community-maintained MCP servers.This addresses the limited prior examination of which description components are included and how frequently they exhibit smells in real-world MCP deployments.Source: RQ-1: To what extent do MCP tools’… · pp. 3–4
- 97.1% of 856 MCP tool descriptions contain at least one smell, while 56% exhibit an Unclear Purpose smell.Most descriptions also contain multiple smell types, especially Unstated Limitations, Missing Usage Guidelines, and Opaque Parameters.Source: RQ-1: To what extent do MCP tools’… · p. 4
- Tool description smells occur in both official and community-maintained MCP servers.The findings indicate that producing high-quality descriptions is challenging for all types of practitioners.Source: RQ-1: To what extent do MCP tools’… · p. 4
RQ-2: How does resolving tool description smells by augmenting all tool description components impact the performance of FM-based agents?
Augmenting all tool-description components significantly improves agent task success and evaluator-level performance, but increases execution cost and sometimes regresses performance. Component ablations further show that shorter, targeted descriptions can preserve effectiveness without sacrificing performance.
RQ-2
- 5.85 percentage points higher task success rate accompanies a 15.12% Average Evaluator improvement, but performance regresses in 16.67% of MCP Universe cases.These gains reflect higher-quality intermediate execution-step completion across domain-model combinations.Source: RQ-2: How does resolving tool… · p. 4
- 67.46% higher median execution steps reveals an accuracy-cost trade-off, with domain-model combinations supporting different operating points.Practitioners can prioritize peak accuracy or lower execution costs according to deployment requirements.Source: RQ-2: How does resolving tool… · p. 4
RQ-3
- No single component combination consistently improves performance across all domains and models, while shorter targeted descriptions can achieve statistically equivalent performance.Removing the Examples component does not statistically degrade performance across all domain-model combinations, enabling lower-cost, domain- and model-specific descriptions.Source: RQ-2: How does resolving tool… · pp. 4–5
Contributions
- The study contributes a structured scoring rubric, an automated FM-based smell scanner, an all-component description augmentor, and a runtime tool-description router.These artifacts support evaluating description quality, resolving smells, and selecting effective description variants without changing MCP server code.Source: RQ-2: How does resolving tool… · p. 5
2 A Motivational Example
A Yahoo Finance MCP example shows that ambiguous tool descriptions can cause FM agents to select overly broad data ranges, increasing latency and token costs. Explicit parameter names and date formats redirect behavior toward bounded, more efficient calls and motivate broader questions about MCP description quality, augmentation, and influential components.
Phase 2: The unseen inefficiency
- Ambiguous guidance caused the FM to use broad multi-year periods for narrow time-bounded questions, inflating tool responses and downstream token usage.The agent called get_historical_stock_prices with a broad period despite the user’s narrow time range, slowing responses and increasing costs.Source: §2 · p. 5
Phase 3: Root cause in the description
- The root cause was a description that mentioned start and end without naming them as parameters or specifying their data format.Without concrete names and formatting guidance, the FM could not reliably construct bounded ranges and defaulted to broader period values; the paper characterizes this as a specification problem rather than a model bug.Source: §2 · p. 5
Phase 4: A small fix that changes behavior
- Explicit start_date and end_date arguments with yyyy-mm-dd formatting led the FM to fetch only needed windows, reducing data volume, latency, and token cost.The clarified description also improved answer relevance for time-scoped queries; Figure 2 compares the ambiguous original and clarified forked descriptions.Source: §2 · p. 6, Figure 2 · p. 6
Phase 5: The challenge
- The example motivates investigating MCP description-smell prevalence across server types and whether systematic augmentation improves agent performance with trade-offs.It also raises the question of which description components most affect performance and whether a generalizable “golden rule” exists across servers and use cases.Source: §2 · pp. 6–7
3 Background and Related Work
MCP standardizes how FM-based agents discover and invoke external tools, making natural-language tool descriptions central to runtime tool selection and use. Prior research has studied smells, refactoring, and MCP-agent evaluation, but has not empirically examined tool-description smells or whether augmenting descriptions changes agent outcomes.
3.1 Model Context Protocol (MCP)
- MCP uses a client-server architecture in which clients discover server-exposed tools through reflection, then pass names, descriptions, and input schemas to FMs for tool selection and invocation.Servers may run locally or remotely, expose multiple tools, resources, and prompts [42], and communicate through JSON-RPC with transports including stdio and streamable HTTP [38].Source: §3.1 · p. 7
- MCP tool descriptions are the primary handshaking interface with FMs, so they should state a tool’s purpose, usage guidance, and relevant caveats or examples [86].The MCP client embeds descriptions with the user query; the FM selects tools, infers arguments, executes calls, and synthesizes answers from returned data.Source: §3.1 · pp. 7–8
3.2 Studies on Smells
- Research has expanded smell analysis from source code to architecture, tests, requirements, FM-generated artifacts, data, and prompts, while MCP work has reported server code smells but not specification-level smells.This study treats tool descriptions as first-class artifacts and examines their impact on agent performance.Source: §3.2 · p. 8
3.3 Refactoring the Smells and Optimization
- Prior refactoring research spans static analysis, machine learning, FM-based correction, and prompt optimization, but no study has investigated optimizing or refactoring MCP tool descriptions.The study adapts FM-based augmentation to improve MCP tool-description quality.Source: §3.3 · p. 9
3.4 Evaluating MCP-Enabled AI Agents
- Existing MCP-agent benchmarks evaluate tool-use behavior across realistic domains and trajectories, but keep server specifications static and provide no evidence on whether changing descriptions improves or degrades outcomes.This study evaluates augmented descriptions and uses ablations to isolate individual description components’ effects on downstream performance.Source: §3.4 · pp. 9–10
4 Methodology … 4.3 Scanning Smells in the Tool Descriptions
The methodology builds a six-component rubric for evaluating MCP tool descriptions, collects 856 tools from 103 servers, and uses multi-model scoring with human validation to derive description smells. Smells are assigned when aggregated component scores fall below the minimum viable threshold of 3.
4.1 Rubric Development
- The study operationalizes tool-description quality through six independently scored components on a 5-point Likert rubric, with Score 3 as the minimum viable threshold.The components include Purpose, Guidelines, Limitations, Parameter Explanation, Length and Completeness, and Examples; requirement-like components define functional contracts, while prompt-like components guide FM behavior.Source: §4.1 · p. 10, §4.1.3 · pp. 11–12, §4.1.4 · p. 13
4.1.1 Official MCP documentation search; 4.1.2 LLM-assisted survey for community guidelines; 4.1.3 Open coding to identify components of tool description
- Mean Jaccard similarity was 0.92 across independent coding of 15 community and official sources, supporting the six-component taxonomy.The sources combined Anthropic’s official guidance with practitioner and academic recommendations gathered through LLM-assisted deep research.Source: §4.1.3 · p. 11, §4.1.2 · p. 11, §4.1.1 · p. 11
4.1.4 Scoring & smell derivation
- Scores below 3 deterministically map to six component-specific smells, eliminating the need for an additional heuristic or classifier.The taxonomy includes Unclear Purpose, Missing Usage Guidance, Unstated Limitation, Opaque Parameters, Underspecified or Incomplete, and Exemplar Issues.Source: §4.1.4 · pp. 13–14, Figure 6 · p. 14
4.2 Tool Description Collection; 4.2.1 MCP server curation
- The dataset contains 856 tools across 103 MCP servers, curated from four studies meeting real-world, empirical, and public-release criteria.The collection includes 23 officially maintained servers and 80 community-maintained servers.Source: §4.2.1 · pp. 14–15
4.2.1 MCP server curation
- A lightweight reflection-based MCP client extracts names, descriptions, and input schemas through tools/list requests across heterogeneous implementations without static source-code analysis.This approach supports servers implemented in different languages and includes proprietary systems.Source: §4.2.2 · p. 15
4.3 Scanning Smells in the Tool Descriptions; 4.3.1 Rubric-based evaluation prompt design; 4.3.2 Multi-model LLM-as-Jury evaluation
- The FM-based scanner assigns ordinal scores from 1 to 5 for all six components using a rubric prompt, controlled environment, and multi-model LLM-as-jury design.Three models from distinct families independently score every description to reduce model-specific bias and improve generalizability.Source: §4.3 · p. 15, §4.3.1 · p. 15, §4.3.2 · p. 15
4.3.2 Multi-model LLM-as-Jury evaluation
- Good reliability was observed in 5 of 6 components, while Examples showed moderate reliability across the three LLM raters.The study quantifies agreement with ICC(2,1) across all 856 descriptions and attributes lower agreement for Examples to stylistic or model-specific variation.Source: §4.3.2 · pp. 15–16, Table 1 · p. 16
4.3.3 Manual Validation for Rubric-Based Scoring; 4.3.4 Smell identification
- Weighted Cohen’s Kappa ranged from 0.72 to 0.89 across components, showing substantial agreement between LLM-jury scores and human assessments of 90 sampled tools.The three-model scores are averaged per component, and an average below 3 triggers the corresponding smell; for example, Purpose scores of 3, 2, and 3 yield an average of approximately 2.7 and an Unclear Purpose smell.Source: §4.3.3 · p. 16, §4.3.4 · pp. 16–17, Table 2 · p. 17
4.4 Resolving Smells via Tool Component Augmentation · 4.5 Evaluating the Augmented Tool Descriptions
The paper augments MCP tool descriptions through rubric-guided FM refinement grounded in execution traces, then evaluates configurable component combinations on MCP-Universe using execution-based metrics and cost-aware experimental adaptations. The benchmark subset is broadly comparable to the remaining corpus in description quality, while the supplied passages establish the evaluation design rather than final performance gains.
4.4 Resolving Tool Component Augmentation · 4.4.1 Initial augmentation of components.
- The semi-automated augmentor combines rubric-based guidance with an FM to preserve tool intent while improving Purpose, Guidelines, Limitations, Parameter Explanation, and Length.GPT-4.1-mini produces initial augmented descriptions from original descriptions and input schemas; Examples are deferred because execution-free generation risks hallucination.Source: §4.4 · p. 17, §4.4.1 · p. 17
4.4.1 Initial augmentation of components. · 4.4.2 Generation of the "Examples and Limitations" components.
- Execution traces ground Examples and Limitations that the FM cannot reliably infer from prompts alone, reducing risks of hallucination and factual inconsistency.The workflow invokes tools on generated tasks, records arguments, responses, and outcomes, then extracts grounded components from those traces.Source: §4.4.1 · p. 17, §4.4.2 · p. 18
4.4.2 Generation of the "Examples and Limitations" components.
- At least two tasks per tool cover successful and failing behavior, with 1–3 additional edge-case tasks selected according to tool complexity.These traces can expose empty responses, errors, multi-step behavior, response-size constraints, and rate limits.Source: §4.4.2 · p. 18
4.4.3 Automated Task Generation for Scalable Augmentation.
- The scalable workflow generates difficulty-prioritized tasks, executes them through a ReAct agent, and uses a GPT-4.1 Jury LLM to extract grounded Examples and Limitations automatically.It removes manual task transfer and log extraction while preserving trace-based grounding and supporting larger MCP servers.Source: §4.4.3 · p. 18
4.4.4 Final consolidation of all components.
- Final augmented descriptions contain five explicit fields—Purpose, Guidelines, Limitations, Parameter Explanation, and Examples—while Length and Completeness remain meta-quality dimensions rather than separate fields.These five-component descriptions are used in subsequent evaluations and ablations.Source: §4.4.4 · p. 19
4.5 Evaluating the Augmented Tool Descriptions · 4.5.1 Benchmark adoption. · 4.5.2 Tool Description Router.
- MCP-Universe evaluates 231 real-world tasks across six domains and 202 tools using execution-based evaluators that verify formatting, static content, and temporal correctness.The extended Tool Description Router dynamically switches original or augmented descriptions and assembles selected components for ablations.Source: §4.5.1 · p. 19, §4.5.2 · pp. 19–20
4.5.1 Benchmark adoption.
- Purpose and Length & Completeness differed significantly between MCP-Universe tools and the remaining corpus, but effect sizes were small or negligible, indicating broadly comparable description quality.The other four components showed no statistically significant differences after Bonferroni correction.Source: §4.5.1 · p. 19, Table 3 · p. 20
4.5.3 Full Rubric Evaluation. · 4.5.4 Ablation Study.
- Evaluation compares Success Rate, Average Evaluator score, and Average # of Steps across models, using hybrid baseline comparisons and adaptations for context, domain, cost, and API constraints.Qwen3-Next-80B-A3B-Instruct receives paired baseline and treatment runs for significance testing, while ablations test Examples and Purpose-centered component combinations.Source: §4.5.3 · pp. 20–21, §4.5.4 · p. 21
5 Results
The results assess how MCP tool-description smells and augmentation affect agent performance, execution cost, and task coverage. They also examine component ablations, showing that context-dependent pruning can retain reliability with lower overhead while revealing differences in component effectiveness and coverage.
5.1 RQ-1: To what extent do MCP tools’ descriptions contain smells?
- 89.8%: Unstated Limitations is the most prevalent smell, followed by Missing Usage Guidelines at 89.3% and Opaque Parameters at 84.3%.Underspecified or Incomplete descriptions affect 79.1% of tools, Exemplar Issues affect 77.9%, and Unclear Purpose affects 56%.Source: §5.1 · p. 22, Figure 7 · p. 23
- 2.9%: only this share of tool descriptions is fully smell-free, declining from 44.0% for Purpose alone and 10.4% with Guidelines to 7.5% for Purpose, Guidelines, and Limitations.The decline across rubric combinations indicates that many descriptions are incomplete across required components.Source: §5.1 · pp. 22–23, Table 4 · p. 23
- No statistically significant differences appear between official and community servers across rubric components: all raw p-values exceed 0.17 and all Bonferroni-corrected p-values equal 1.0.Both server groups show low-quality tool descriptions across all rubric components.Source: Table 5 · p. 23, §5.1 · pp. 23–24
5.2 RQ-2: How does resolving tool description smells by augmenting all tool description components impact the performance of FM-based agents?
- 5.85 percentage points: full description augmentation significantly increases median task success rate across models and domains, although 16.67% of cases regress.The improvement is significant across the full benchmark (p = 0.02), with augmented descriptions outperforming baselines in 54.17% of cases.Source: §5.2 · pp. 25, 26, 29, Table 7 · p. 26
- 15.12%: augmentation increases Average Evaluator Score, indicating greater partial goal completion, while median execution steps rise 67.46%.The AE increase is statistically significant (p < 0.01); three of four models show significant AS increases (p < 0.001), although Qwen3-Next-80B-A3B-Instruct reduces AS while improving SR and AE.Source: Table 8 · p. 27, §5.2 · pp. 27, 29
5.3 RQ-3: How do different components of the augmented tool description impact the performance of FM-based agents?
- No single component combination is universally best: Purpose plus Guidelines leads in Finance with GPT-4.1, full augmentation leads in Location Navigation with GPT-4.1, and removing Examples modestly helps Repository Management with Qwen3-Coder-480B-A35B.Best combinations are strongly associated with full-augmentation outcomes, with signed 𝜙 coefficients of approximately 0.5–0.9, suggesting domain-specific pruning can preserve reliability while reducing token usage and latency.Source: §5.3 · pp. 31–32, Table 10 · p. 32, Table 11 · p. 32
- P+G+L+PEx achieved the highest median SR, 21.21%, marginally surpassing FR at 21.10%, while remaining stable across approximately 60% of domain-model pairs.This indicates that removing Examples does not hurt and may slightly improve cross-domain consistency.Source: §5.3 · p. 32
- BC and FR solve overlapping but distinct task sets, with neither configuration fully subsuming the other across five domain-model combinations.In Location Navigation with GPT-4.1, they shared nine successfully solved tasks but diverged on eight, indicating distinct reasoning trajectories.Source: Figure 12 · p. 33, §5.3 · p. 33
- Cochran’s Q tests yielded p>0.20 across all domain-model combinations, showing no statistically significant success-rate differences among FR, BC, and P+G+L+PEx.Thus, removing Examples does not materially affect task success rates, supporting Anthropic’s guidance while contrasting with traditional few-shot benefits.Source: §5.3 · pp. 32–33
- Combining FR with reduced-component retries could raise Location Navigation success to 37.8%, but the hybrid approach increases execution cost through additional steps and latency.This creates a potential accuracy-oriented strategy for mission-critical scenarios where high success rates outweigh cost.Source: §5.3 · p. 33
- No single “golden” component combination exists: effectiveness varies by model and domain, while small pruned sets can preserve logical reliability and offer cost-efficient alternatives to fully augmented descriptions.These reduced configurations have task-level outcomes strongly associated with fully augmented descriptions, depending on the domain and model.Source: §5.3 · p. 34
6 Discussion
Source-code grounding improves rubric-scoring reliability and augmentation faithfulness, especially for parameter explanations, but does not guarantee better task performance. Rubric-defined smells are useful risk indicators whose effects depend on execution context, tool complexity, and model familiarity.
6.1 Impact of Source Code Grounding in Smell Detection
- Source-code grounding raised Parameter Explanation’s median rubric score from 1.0 to 2.0, with p< 0.001 and the largest effect size, r= 0.60.Grounding changed evaluator judgments and generally increased scores for three of six components, helping recover implementation details absent or unclear in descriptions.Source: §6.1 · pp. 34–35, Table 12 · p. 35
6.2 Impact of Source Code Grounding in Augmentation
- Source-code grounding increased Parameter Explanation’s median faithfulness from 0.71 to 1.00 and significantly improved faithfulness for four components.It reduces hallucinated or unsupported details, but grounded descriptions solved 26 tasks versus 35 ungrounded tasks across 92 tasks, with no significant difference (p= 0.064).Source: §6.2 · p. 36, Table 14 · p. 36
6.3 Impact of Rubric Components in Potential Smell Detection
- 5.85 percentage points higher task success and 15.12% higher Average Evaluator Score (AE) followed smell resolution, but 16.67% of cases regressed.Performance improved in 54.17% of cases, remained unchanged in 29.16%, and augmentation increased execution cost, showing that rubric components provide a context-dependent quality signal.Source: §6.3 · p. 37
- Compact descriptions can match or outperform fully augmented versions, while missing components are potential risk indicators rather than deterministic predictors of failure.Examples did not significantly degrade performance when removed, whereas fully augmented descriptions may be more beneficial for unfamiliar domains, complex tasks, or subtle invocation constraints.Source: §6.3 · p. 37
7 Implications
The implications call for treating tool descriptions as engineered, dynamically managed artifacts across the MCP ecosystem. Developers, maintainers, users, and researchers should balance behavioral gains against context, execution, and token costs.
7.1 Implications for MCP Developers
- MCP developers should make rubric-based smell detection a release or CI gate and prioritize high-leverage components before selectively adding costly descriptive detail.Unclear Purpose affects 56% of tools, while 89.3% lack Usage Guidance; Purpose alone leaves 44% smell-free, versus 2.9% across all five major components.Source: §7.1 · p. 38
7.2 Implications for Ecosystem Maintainers
- Ecosystem maintainers should structure descriptions into component fields, support runtime selective loading, and provide registry-wide smell scans, quality scores, and warnings.Structured fields could enable agents to load Purpose first and defer Guidelines or Examples; registry diagnostics are justified because smells occur across official and community servers.Source: §7.2 · pp. 38–39
7.3 Implications for MCP Users
- MCP users should customize descriptions client-side, use augmentation to enable smaller models where reliable, and enforce resource caps matched to domain-specific overhead.Augmentation raises median Success Rate by 5.85 percentage points but Average Steps by 67.46%, requiring higher budgets where gains justify costs and compact profiles where they do not.Source: §7.3 · pp. 39–40
7.4 Implications for Researchers
- Researchers should test whether precise descriptions improve dynamic tool discovery, planning, orchestration-code reliability, and progressive-disclosure skill invocation without reproducing multi-step resource penalties.Relevant directions include active or reactive tool search, programmatic execution, and ablations identifying the smallest metadata signals that reliably trigger skill use.Source: §7.4 · p. 40
8 Threats to Validity
The study’s validity is limited by corpus and benchmark coverage, context-dependent augmentation, construct choices in smell detection, and baseline or instrumentation differences. The authors mitigate these threats through diverse sampling, multi-model and human validation, matched evaluations, and robustness checks.
8.1 External validity
- The corpus may underrepresent unevaluated, proprietary, or recent servers, while performance results cover only 202 tools from 18 MCP servers and a heterogeneous subset of models.The full corpus contains 856 tools across 103 servers; MCP-Universe provides diverse tasks and deployment settings, but remains a strict subset. Dynamic reflection keeps the pipeline applicable to open- and closed-source servers.Source: §8.1 · pp. 40–41
- Context limits require partial augmentation for Qwen3-Next-80B-A3B-Instruct and Browser Automation, so benefits may be unrealizable for constrained models or verbose domains.Parameter Explanation and Examples are omitted in specified settings; schemas partly compensate for missing parameter explanations, while ablations suggest omitting Examples usually has limited impact.Source: §8.1 · p. 41
8.2 Construct validity
- Smell thresholds and FM scoring introduce subjectivity and model bias, while source-code grounding excludes proprietary, unsupported-language, unconventional, and upstream-API functionality.The study uses structured Likert-style scoring, a three-family LLM jury, inter-rater reliability, human agreement checks, and code-plus-schema grounding where available.Source: §8.2 · p. 42
8.3 Internal validity
- Baseline comparisons for three models rely on aggregate prior metrics, and stochastic workflows may confound augmentation effects; matched baseline and augmented runs address this for Qwen3-Next-80B-A3B-Instruct.Paired statistical tests assess changes in SR, AE, and AS for the newly executed model.Source: §8.3 · p. 42
- Web Searching results may reflect different search APIs, while the 25-run ablation samples selected combinations and always retains Purpose, limiting causal generalization.The new model uses the same search server for both conditions, and the ablation includes both improvements and regressions plus an all-but-Examples configuration.Source: §8.3 · pp. 42–43
9 Conclusion
Across 856 MCP tools, tool descriptions emerge as critical yet under-engineered artifacts: rubric-aligned augmentation improves agent performance but introduces execution-cost and context-sensitivity trade-offs. The findings motivate configurable, component-aware descriptions and cost-aware context management.
- 15.12% increase in the Average Evaluator Score and 5.85 percentage point improvement in task success rates result from augmenting descriptions with all rubric components, although gains require more execution steps and higher costs.These improvements can enable smaller open-weight models to achieve performance parity with larger frontier models, but do not generalize across all domain-model combinations.Source: §9 · p. 43
- Over 97% of MCP tools suffer from description smells, establishing tool descriptions as a critical but under-engineered artifact across the ecosystem.The study analyzed 856 tools from official and community-maintained servers using rubric-guided augmentation, benchmarking, and controlled studies.Source: §9 · p. 43
- Compact combinations of high-impact description components can preserve behavioral alignment with fully augmented descriptions while reducing cost overhead, motivating configurable and cost-aware protocol designs.Future work should examine impacts on dynamic tool search, code-mode MCP execution, tool retrieval, and orchestration.Source: §9 · p. 43
A Appendix · A.1 Prompts used by the LLM-Jury
The LLM-Jury prompt evaluates tool descriptions through component-level scoring and an overall quality assessment, supplemented by justification and improvement recommendations.
A.1 Prompts used by the LLM-Jury
- The jury scores each tool-description component on a 1-to-5 scale.Source: Judge Tool Description Quality Using a… · p. 49
- It also assigns an overall quality score from 0 to 100.Source: Judge Tool Description Quality Using a… · p. 49
- For each assessment, the prompt requests a justification and recommendations for improving the tool description.Source: Judge Tool Description Quality Using a… · p. 49
Judge Tool Description Quality Using a Six-Component Rubric.
The rubric evaluates MCP tool descriptions on six dimensions using a 1–5 scale, then labels descriptions Good or Bad based on threshold and completeness rules.
Six-component rubric
- The rubric defines five dimensions: purpose, usage guidelines, limitations, parameter explanation, and examples-versus-description balance.Source: Judge Tool Description Quality Using a… · pp. 49–50
- Purpose scores range from unclear or missing to precise explanations of function, behavior, and return data.The rubric also evaluates explicit use and non-use cases, disambiguation, caveats, scope boundaries, and constraints.Source: Judge Tool Description Quality Using a… · pp. 49–50
- Parameter explanations receive higher scores when every input includes its type, meaning, behavioral effect, and required or default status.Examples should supplement rather than replace self-sufficient prose, while length and completeness reward substantive coverage across the description.Source: Judge Tool Description Quality Using a… · p. 50
Labeling and output
- A description is labeled Bad if any rubric dimension scores below 3 or examples replace the description; Good requires all six scores to be at least 3.The output includes per-dimension scores, a Good/Bad label, a one-sentence reason, and weak areas with scores of 3 or lower.Source: Judge Tool Description Quality Using a… · pp. 50–51
A.2 Prompts used by Tool Description Augmentor
The augmentor uses one prompt to generate realistic, single-call benchmark tasks and another to synthesize execution evidence into factual tool documentation. Both prompts constrain outputs to concise, grounded content and explicitly address edge cases and limitations.
A.2.1 Task Generation Prompt.
- The task-generation prompt asks for realistic benchmark questions testing correct use of a specific MCP tool.It frames the model as an expert benchmark designer for AI-agent evaluation.Source: §A.2.1 · p. 51
- Each generated task must have one unambiguous answer obtainable through a single tool call and use concise, conversational wording.Tasks should resemble real user questions rather than formal tool-invocation requests.Source: §A.2.1 · p. 51
- Tasks must provide all required input values and vary realistic parameters across tasks to prevent agent guesswork and repetition.Examples of varying values include dates, tickers, names, units, and other parameter values.Source: §A.2.1 · p. 51
- The generator defines medium, hard, and very hard tasks by escalating from normal usage to edge cases, limits, ambiguity resolution, and failure boundaries.Even the most demanding tasks must remain solvable with a single tool call.Source: §A.2.1 · p. 51
- The task-generation output is restricted to a JSON object containing difficulty, question, exact correct answer, and a one-sentence rationale.The prompt forbids prose and markdown fences and specifies the required schema.Source: §A.2.1 · p. 51
A.2.2 Jury LLM Prompt.
- The jury LLM converts observed tool arguments and responses or errors into a concise additional_description that helps agents use the tool correctly.The documentation must be based on exact execution evidence rather than unsupported assumptions.Source: §A.2.2 · p. 52
- The synthesized documentation must contain labelled Examples and Limitations sections, with examples showing exact arguments and responses and limitations reporting only observed constraints or gotchas.Examples should not invent scenarios, while limitations may cover formats, ranges, errors, surprising behavior, and other tool-specific findings.Source: §A.2.2 · p. 52
- The jury prompt requires plain, direct English, prohibits unsupported facts and verbatim repetition of the tool description, and returns documentation text without a JSON wrapper.It also forbids a markdown heading for the tool name itself.Source: §A.2.2 · p. 52
Related papers
This paper cites
Cited by
- Right Numbers, Wrong Questions: Evaluating LLMs on Underspecified Financial Queries2026
- ProbeID: Verifying MCP Tool Identity for LLM Agents under Registry Drift2026
- Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution2026
- Tool-calling retrieval versus vector RAG for a small Greek--English knowledge base: accuracy and robustness to how users type Greek2026
- STEM Agent: A Self-Adapting, Tool-Enabled, Extensible Architecture for Multi-Protocol AI Agent Systems2026