PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions

Mohammed Mehedi Hasan, Hao Li, Gopi Krishnan Rajbahadur, Bram Adams, Ahmed E. Hassan

arXiv:2602.14878v3cs.SEcs.ET

TL;DR

MCP agents depend on tool descriptions, yet defects in these descriptions can undermine reliable tool selection and argument construction. This paper evaluates description smells across the MCP ecosystem, formalizes a rubric-based scanner, and finds that augmentation improves performance overall but increases execution costs and sometimes regresses results.

  • Problem

    Evidence on the prevalence and consequences of defective MCP tool descriptions is limited, despite their role in guiding tool selection and argument construction.

  • Method

    The paper identifies six description components, builds a scoring rubric, and formalizes component-specific smells for empirical analysis.

  • Results

    5.85 percentage points higher task success accompanied 15.12% higher Average Evaluator Score, while 16.67% of cases regressed and execution cost increased.

  • Takeaways & Limitations

    Tool descriptions are configurable engineering artifacts requiring cost-aware and context-sensitive augmentation rather than uniform expansion.

  • Takeaways & Limitations

    The supplied evidence bounds the task-generation example to the get_historical_stock_prices tool in the yfinance MCP server and three specified retrieval tasks.

Abstract

from arXiv · show

The Model Context Protocol (MCP) introduces a standard specification that defines how Foundation Model (FM)-based agents should interact with external systems by invoking tools. However, to understand a tool's purpose and features, FMs rely on natural-language tool descriptions, making these descriptions a critical component in guiding FMs to select the optimal tool for a given (sub)task and to pass the right arguments to the tool. While defects or smells in these descriptions can misguide FM-based agents, their prevalence and consequences in the MCP ecosystem remain unclear. Hence, we examine 856 tools spread across 103 MCP servers empirically, assess their description quality, and their impact on agent performance. We identify six components of tool descriptions from the literature, develop a scoring rubric utilizing these components, and then formalize tool description smells based on this rubric. By operationalizing this rubric through an FM-based scanner, we find that 97.1% of the analyzed tool descriptions contain at least one smell, with 56% failing to state their purpose clearly. While augmenting these descriptions for all components improves task success rates by a median of 5.85 percentage points and improves partial goal completion by 15.12%, it also increases the number of execution steps by 67.46% and regresses performance in 16.67% of cases. These results indicate that achieving performance gains is not straightforward; while execution cost can act as a trade-off, execution context can also impact. Furthermore, component ablations show that compact variants of different component combinations often preserve behavioral reliability while reducing unnecessary token overhead, enabling more efficient use of the FM context window and lower execution costs.

1 Introduction

MCP tool descriptions are a critical design surface: they guide FM-based agents’ tool selection, argument formation, and orchestration while functioning as both behavioral specifications and contextual instructions. This study addresses the lack of systematic evidence about their quality and downstream effects by empirically analyzing MCP servers and augmenting descriptions to evaluate agent performance.

  • Tool descriptions guide FM-based agents’ tool selection, parameterization, and multi-step orchestration by conveying intended functionality, constraints, and usage cues.
  • Defective, underspecified, or misleading descriptions can cause wrong tool selection, invalid or suboptimal arguments, unnecessary interaction steps, and reduced system reliability.
  • Tool descriptions have a dual role as requirement-like specifications of expected behavior and parameter constraints and prompt-like instructions shaping contextual reasoning and decision-making.
  • The prevalence, distribution, and downstream performance impact of MCP tool-description smells had not been systematically investigated despite related work on tool calling, tool sets, security, and writing guidelines.
  • 103 MCP servers and 856 tools form the study dataset, whose descriptions are scanned with a structured quality rubric before FM-based smell repair and augmentation are evaluated on agent performance.

RQ-1: To what extent do MCP tools’ descriptions contain smells?

MCP tool descriptions commonly contain quality smells: 97.1% of the 856 analyzed descriptions exhibit at least one smell, while 56% have an Unclear Purpose smell. Multiple smell types are prevalent across both official and community-maintained servers.

  • Findings: 97.1% of the 856 MCP tool descriptions contain at least one smell.The majority exhibit multiple smell types.
  • Findings: 56% of the 856 MCP tool descriptions exhibit an Unclear Purpose smell.This indicates that a majority fail to articulate their intended functionality clearly to the FM.
  • Findings: Unstated Limitations, Missing Usage Guidelines, and Opaque Parameters are particularly common smell types.These issues affect descriptions from both official and community-maintained servers.

RQ-2: How does resolving tool description smells by augmenting all tool description components impact the performance of FM-based agents?

Augmenting all tool-description components significantly improves agent task success and evaluator-level performance, but increases execution overhead and regresses performance in some benchmark cases.

  • Performance effects: 5.85 percentage points: augmented descriptions significantly increase task success rate across domain-model combinations.The MCP Universe benchmark also reports regressions in 16.67% of cases.
  • Performance effects: 15.12%: the Average Evaluator score increases, reflecting higher-quality intermediate execution step completion.
  • Performance effects: 67.46%: the average number of execution steps increases, creating a trade-off for the performance gains.
  • Performance effects: 16.67%: augmented descriptions regress performance in cases within the MCP Universe benchmark.

2 A Motivational Example

A finance-assistant example shows how ambiguous MCP tool descriptions can cause broad, inefficient data retrieval, while explicit parameter names and formats improve time-bounded calls. This motivates studying description-smell prevalence, augmentation trade-offs, and the components that most affect agent performance.

  • Scenario: A Yahoo Finance MCP server is used to build a finance assistant for portfolio questions and simple what-if queries.
  • Phase 1: It works: During development, the agent successfully handles simple requests by selecting the historical-price tool and returning its data.
  • Phase 2: The unseen inefficiency: Broad period selections for narrow time-bounded questions inflate tool responses, latency, costs, and downstream FM token usage.The agent calls get_historical_stock_prices with a broad period that expands to multi-year windows.
  • Phase 3: Root cause in the description: Ambiguous descriptions omit explicit start and end parameter names and date formats, causing the FM to default to overly broad period ranges.The original description says to use either the period parameter or start and end without specifying concrete names, data types, or formats.
  • Phase 4: A small fix that changes behavior: Explicit start_date and end_date arguments with yyyy-mm-dd formatting lead to bounded calls that reduce data volume, latency, and token cost while improving answer relevance.
  • Phase 5: The challenge: The example motivates research on MCP description-smell prevalence, augmentation trade-offs, and the components that most affect agent performance across servers and use cases.The questions address distribution across official and community-maintained servers, realistic workflow performance, trade-offs, and whether a generalizable golden rule exists.

3 Background and Related Work

MCP standardizes how foundation-model agents discover and invoke external tools through a client-server protocol, with tool descriptions serving as the primary interface for tool selection and argument inference. Related work studies smells across software and FM artifacts, but this study addresses the gap in empirical analysis and optimization of MCP tool-description smells.

  • Background: MCP uses a client-server architecture in which clients connect to local or remote servers that expose tools, resources, and prompts through JSON-RPC discovery and capability negotiation.Clients enumerate available tools before execution through an initialization handshake and list/get methods.
  • Background: Tool descriptions contain a name, natural-language description, and input schema that MCP clients pass to foundation models for selecting and invoking the correct tool.These descriptions are the primary handshaking interface between MCP servers and foundation models.
  • Background: The MCP client embeds tool descriptions with the user query, after which the model selects a tool, infers arguments, and issues a validated execution request.The client can seek user consent for sensitive actions, execute calls through the appropriate server, and synthesize an answer from returned data.
  • Related Work: Prior research examines smells in code, architecture, tests, requirements, designs, FM-generated artifacts, data, and prompts, while recent MCP work reports code smells in servers.Smells are treated as latent design or process weaknesses that can hinder comprehension, maintenance, or output quality.
  • Related Work: No prior study had empirically investigated MCP tool-description smells or their optimization, while existing benchmarks kept server specifications static.This study evaluates augmented descriptions and uses ablation studies to isolate individual description components’ effects on downstream agent performance.

4 Methodology … 4.3 Scanning Smells in the Tool Descriptions

The study builds a six-component rubric for evaluating MCP tool descriptions, collects 856 tools from 103 servers, and operationalizes smell detection through multi-model FM scoring, aggregation, and thresholding. Manual validation and inter-rater analysis support the reliability of this evaluation methodology.

  • 4 Methodology: The methodology proceeds from rubric development through tool collection and FM-based smell scanning, with each stage described in successive subsections.The study explicitly organizes its methodology into these sequential steps.
  • 4.1 Rubric Development: The rubric evaluates six tool-description components independently on a 5-point Likert scale, with Score 3 defined as the minimum viable threshold.The rubric is designed as a structured, multidimensional framework rather than a qualitative checklist.
  • 4.1.1 Official MCP documentation search.: Official Anthropic MCP documentation and an LLM-assisted survey of community guidelines jointly inform the component framework, which authors derive through independent open coding of 15 sources.The survey uses OpenAI Deep Research with GPT-5.1, while the first two authors manually synthesize recurring recommendations.
  • 4.1.3 Open coding to identify components of tool description.: The six components comprise specification-oriented content—purpose, guidelines, limitations, parameter explanation—and completeness-oriented content involving length and examples.Specification components define functional requirements and execution constraints, while the remaining components address descriptive completeness.
  • 4.1.4 Scoring & smell derivation.: Scores below 3 indicate component deficiencies and directly map to corresponding tool-description smells, eliminating the need for an additional heuristic or classifier.The taxonomy includes smells such as Unclear Purpose, Missing Usage Guidance, Unstated Limitation, and Opaque Parameters.
  • 4.2 Tool Description Collection: The dataset contains 856 tools across 103 MCP servers curated from prior empirical studies meeting real-world, general-purpose, and public-release criteria.A lightweight MCP client dynamically retrieves tool names, descriptions, and input schemas through the tools/list reflection request.
  • 4.3 Scanning Smells in the Tool Descriptions: Three FM families independently score every description using a structured rubric prompt, forming a multi-model LLM-as-jury intended to reduce model-specific bias.The selected models include gpt-4.1-mini, claude-haiku-3.5, and qwen3-30b-a3b.
  • 4.3.2 Multi-model LLM-as-Jury evaluation: Good inter-rater reliability occurs for 5 out of 6 components, while Examples achieves moderate reliability; human scores are additionally compared with aggregated jury scores using weighted Cohen’s Kappa.For smell assignment, the three FM scores are averaged, and any consolidated score below 3 receives the corresponding taxonomy smell.

4.4 Resolving Smells via Tool Component Augmentation · 4.5 Evaluating the Augmented Tool Descriptions

The study develops a semi-automated augmentor that enriches MCP tool descriptions using rubric-guided FM generation and execution traces, then evaluates the augmented descriptions on MCP-Universe through dynamic routing, full-rubric experiments, and component ablations. The evaluation preserves realistic examples and limitations while comparing augmented descriptions with reported baselines under benchmark and model constraints.

  • 4.4 Resolving Smells via Tool Component Augmentation: The semi-automated augmentor combines rubric-based augmentation with an FM to produce refined, comprehensive, and factually consistent descriptions while preserving original meaning and intent.It improves coverage across five rubric components: Purpose, Guidelines, Limitations, Parameter Explanation, and Examples.
  • 4.4.1 Initial augmentation of components: Execution traces ground Examples and Limitations because the FM cannot reliably infer realistic content or complete limitations from prompts alone without execution context.The workflow generates successful, failing, and edge-case tasks to capture tool behavior, response limits, and rate-limit-related constraints.
  • 4.4.3 Automated Task Generation for Scalable Augmentation: The final augmented representation uses five explicit fields and omits a separate Length and Completeness field because that dimension is treated as a meta-quality property.The five fields are Purpose, Guidelines, Limitations, Parameter Explanation, and Examples.
  • 4.5 Evaluating the Augmented Tool Descriptions: The benchmark contains 231 complex real-world tasks across six domains and 202 tools, combining LLM-as-Jury assessment with execution-based evaluation.The benchmark was selected for its real-world scenarios and temporal dynamics.
  • 4.5.1 Benchmark adoption: Purpose and Length & Completeness differ statistically between MCP-Universe tools and the remaining corpus, but their Cliff’s delta values are small or negligible at δ=−0.15 and δ=−0.14.Usage Guidelines, Limitations, Parameter Explanation, and Examples show no statistically significant differences after Bonferroni correction, indicating broad comparability.
  • 4.5.2 Tool Description Router: The Tool Description Router enables runtime selection between original and augmented descriptions and supports assembling descriptions from specified component combinations for ablation studies.It retrieves augmented descriptions from PostgreSQL and can return selected components in a requested order.
  • 4.5.3 Full Rubric Evaluation: Evaluation compares augmented descriptions with reported MCP-Universe baselines using Success Rate, Average Evaluator score, and Average # of Steps.Because per-task baseline outcomes are unavailable, the study uses a hybrid comparison strategy rather than paired statistical tests.
  • 4.5.4 Ablation Study: The ablation study selectively excludes Examples or evaluates pairwise combinations in which Purpose is always included to identify efficient component configurations.Model and context constraints also motivate adaptations, including excluding Parameter Explanation for Qwen3-Next-80B-A3B-Instruct.

5 Results

The results show widespread defects in MCP tool descriptions and significant but uneven benefits from augmentation. Full augmentation improves rubric scores, task and evaluator performance, and cost-accuracy trade-offs, but increases execution effort and can regress outcomes; component combinations are domain-model dependent.

  • Description quality: 89.8% of descriptions contain Unstated Limitations, 89.3% contain Missing Usage Guidelines, and 84.3% contain Opaque Parameters.These are the three most widespread smell categories, indicating missing boundary conditions, invocation guidance, and parameter meaning or behavior.
  • Description quality: 2.9% of MCP tool descriptions are fully smell-free, falling to 7.5% when Purpose, Guidelines, and Limitations are considered together.Purpose alone has a 44.0% smell-free rate, compared with 10.4% for Purpose and Guidelines.
  • Description quality: 97.1% of analyzed descriptions therefore contain at least one smell, while official and community servers show no statistically significant component differences.For the six-component comparison, all raw p-values exceed 0.17 and all Bonferroni-corrected p-values equal 1.0.
  • Augmentation outcomes: 2.7–4.0 points is the median rubric-score improvement after FM-based augmentation, with statistically significant gains across all six components (p < .001).Scores are compared before augmentation (BA) and after augmentation (AA) using the Wilcoxon signed-rank test.
  • Augmentation outcomes: 5.85 percentage points is the median task-success increase, while 15.12% is the Average Evaluator Score increase and 16.67% of cases regress.Agents using augmented descriptions outperform baseline counterparts in 54.17% of cases.

6 Discussion

Source-code grounding makes rubric evaluation and augmented descriptions more reliable by revealing implementation-supported details and reducing hallucinations, especially for parameter explanations. However, grounding does not guarantee better task performance, while compact, task-relevant descriptions may preserve or improve performance at lower context cost.

  • Source-code grounding: Source-code grounding increased the median Parameter Explanation score from 1.0 to 2.0, with p< 0.001 and the largest effect size (r= 0.60).Implementation evidence helps evaluators recognize parameter details recoverable from code but absent or unclear in original descriptions.
  • Source-code grounding: Source-code grounding improved augmented-description faithfulness overall, with the strongest gain for Parameter Explanation, whose median increased from 0.71 to 1.00.Statistically significant gains also occurred for Limitations, Purpose, and Guidelines.
  • Task-level performance: 26 grounded-description tasks were solved versus 35 with ungrounded descriptions, with no statistically significant difference (p= 0.064).The comparison reexecuted 92 tasks from Finance, Repository Management, and 3D Design.
  • Implications: Grounding is best viewed as a reliability safeguard rather than a guaranteed performance optimization, with benefits potentially greater for rare, proprietary, or domain-specific tools.Relevant future factors include tool complexity, model familiarity, task requirements, context budget, and prior exposure to the tool or domain.
  • Task-level performance: Augmentation improved task success by 5.85 percentage points and Average Evaluator Score by 15.12%, but 16.67% of cases regressed.Performance improved in 54.17% of cases and remained unchanged in 29.16%.
  • Compact augmentation: Compact descriptions can match or outperform fully augmented descriptions when they preserve the tool’s core purpose and relevant usage constraints.The Examples ablation most strongly supports this divergence, indicating that not every missing component has equal operational severity.

7 Implications

The implications call for systematic quality control, selective description augmentation, structured metadata, client-side customization, and resource-aware deployment. They also identify research opportunities for integrating description quality with dynamic discovery and progressive disclosure.

  • MCP developers: 56% of tools suffer from Unclear Purpose and 89.3% lack Usage Guidance, motivating rubric-based smell detection in MCP developers’ review or CI pipelines.The paper argues that these smell rates make ad hoc writing practices insufficient and descriptions should be treated as first-class engineering artifacts.
  • MCP developers: 44% of tools are smell-free on Purpose alone, but only 2.9% are smell-free across all five major components, supporting targeted high-leverage fixes.Compact component combinations can preserve or improve performance, so developers need not fix every aspect simultaneously.
  • MCP developers: FM-based augmentation lifts median description scores from the 1-2 range to nearly 5.0 across components, but may create verbose descriptions and context-window costs.The paper recommends treating augmentation as refinement while weighing scale, costs, and manual or semi-automated alternatives.
  • Protocol and registry maintainers: Structured component fields would let agents request Purpose first and lazily load guidelines or examples, reducing token overhead through dynamic context management.Protocol maintainers are encouraged to extend the monolithic description field and registries to provide rubric-based smell detection and quality scoring.
  • MCP users: Client-side description overrides can be applied at runtime without modifying server code, making descriptions a mutable, cost-effective alternative to immediately choosing larger frontier models.The smaller Qwen3-Next-80B-A3B-Instruct achieved parity with or surpassed Qwen3-Coder-480B-A35B in Finance, Repository Management, and Location Navigation when descriptions were augmented.
  • MCP users: MCP users should set explicit resource caps and use compact configurations where gains are small or negative, while allowing higher caps when success-rate gains justify overhead.The passages identify Finance and Repository Management as examples supporting higher caps, and Web Searching as an example favoring lower caps or compact configurations.
  • Researchers: Researchers should test whether augmented descriptions improve dynamic tool-discovery recall and orchestration-code correctness without the resource penalties of traditional multi-step loops.The role of description quality within these efficiency mechanisms remains empirically unexplored.
  • Researchers: Researchers should extend component ablations to progressive-disclosure mechanisms such as agent skills and test whether under-specified metadata causes models to ignore available skills.Agent skills expose minimal metadata and load full instructions on demand to reduce context usage while enabling autonomous invocation.

8 Threats to Validity

The study’s validity is constrained by dataset coverage, benchmark and context-window limitations, subjective smell scoring, implementation-specific source-code grounding, and nonuniform or costly evaluation settings. Additional threats arise from alternative search infrastructure and limited ablation coverage.

  • Dataset coverage: The 856-tool, 103-server dataset may underrepresent proprietary internal deployments and recently introduced or undocumented MCP servers.It nevertheless spans community-maintained, officially managed open-source, and some officially managed closed servers.
  • Evaluation scope: Performance evaluation covers only 202 tools from 18 MCP servers in MCP-Universe, a strict subset of the 856-tool corpus, and omits some models because of benchmark cost.Thus, the performance results do not cover the full corpus or all models included in MCP-Universe.
  • Evaluation scope: Context-window limits forced component exclusions for Qwen3-Next-80B-A3B-Instruct and Browser Automation, so some augmented evaluations are partial.The excluded components were Parameter Explanation and Examples for Qwen3-Next-80B-A3B-Instruct, and Examples for Browser Automation.
  • Construct validity: Smell identification depends on a subjective score-three threshold and FM-based scoring that is sensitive to prompt design and model-specific preferences.These choices create construct-validity concerns even though the study sought to reduce evaluation subjectivity.
  • Construct validity: Source-code grounding is unavailable for proprietary MCP servers and currently supports only Python, JavaScript/TypeScript, and Go servers.Grounding therefore introduces implementation-specific threats and depends on publicly available codebases.
  • Experimental design: Baseline comparisons for three models rely on aggregate prior metrics, while non-deterministic workflows and an alternative Google Search server may affect observed differences.The ablation study also retains Purpose in all 25 runs and omits other component permutations because of computational cost and time.

9 Conclusion

The study establishes MCP tool descriptions as critical yet under-engineered artifacts that substantially affect FM-based agent execution. Rubric-aligned augmentation improves agent performance, while motivating structured, component-aware, cost-aware protocol designs.

  • Empirical characterization: 856 tools across official and community-maintained servers were analyzed to characterize MCP tool descriptions and their effects on FM-based agent execution.The study combined rubric-guided augmentation, benchmarking, and controlled studies.
  • Description quality: Over 97% of MCP tools suffer from ecosystem-wide description smells.The finding supports treating tool descriptions as configurable engineering artifacts.
  • Performance impact: 15.12% increase in the Average Evaluator Score and 5.85 percentage point improvement in task success rates result from rubric-aligned descriptions with all components.The augmentation also enabled smaller open-weight models to achieve performance parity with larger frontier models.
  • Implications and future work: The findings motivate structured, component-aware protocol designs supporting dynamic, cost-aware context management in MCP-enabled agents.Future work should examine dynamic tool search, code-mode MCP execution, and effects on tool retrieval and orchestration.

A Appendix · A.1 Prompts used by the LLM-Jury

The appendix presents the LLM-Jury prompt for evaluating tool descriptions. The jury scores each component, assigns an overall quality score, explains its assessment, and recommends improvements.

  • A.1 Prompts used by the LLM-Jury: The LLM-Jury evaluates tool descriptions using a structured grading prompt.The prompt explicitly frames the task as grading a tool description.
  • A.1 Prompts used by the LLM-Jury: Each description component receives a score from 1 to 5.The prompt instructs the jury to score each component individually.
  • A.1 Prompts used by the LLM-Jury: The jury also provides an overall quality score from 0–100.This score complements the component-level ratings.
  • A.1 Prompts used by the LLM-Jury: The evaluation requires a justification for the assigned scores.The prompt explicitly requests a justification.
  • A.1 Prompts used by the LLM-Jury: The jury must provide recommendations for improving the tool description.Improvement recommendations are a required output of the prompt.
  • A.1 Prompts used by the LLM-Jury: The prompt combines component-level scoring with an aggregate quality assessment.It requests both per-component scores and an overall quality score.
  • A.1 Prompts used by the LLM-Jury: The requested outputs cover ratings, explanation, and actionable recommendations.The prompt asks for scores, a justification, and improvement recommendations.

Judge Tool Description Quality Using a Six-Component Rubric.

Tool descriptions are judged on six components scored from 1–5: purpose, usage guidelines, limitations, parameter explanation, examples-versus-description balance, and length and completeness. The rubric also classifies descriptions as Bad when any dimension scores below 3 or examples replace prose, and as Good only when all dimensions score at least 3 and all component requirements are satisfied.

  • Rubric Components: Six components receive scores from 1–5: purpose, usage guidelines, limitation, parameter explanation, examples balance, and length completeness.The rubric evaluates what the tool does, when to use it, caveats, input clarity, examples’ role, and substantive coverage.
  • Purpose: Purpose scores 5/5 when function, behavior, and return data are precise, but 1/5 when the purpose is unclear or missing.Intermediate scores range from basic explanations lacking behavioral details to explanations with only minor ambiguity.
  • Usage Guidelines: Usage guidelines score 5/5 when they state appropriate and inappropriate use cases and disambiguate ambiguous tool names, while 1/5 means no guidance.The rubric also considers whether usage context is explicit or merely implied.
  • Limitations: Limitations score 5/5 when scope boundaries, non-returned outputs, and important constraints are clear, while 1/5 means no caveats are mentioned.Lower scores reflect vague, incomplete, minimal, or implied limitation statements.
  • Parameter Explanation: Parameter explanation scores 5/5 when every input’s type, meaning, behavioral effect, and required or default status are explained; 1/5 means parameters are unexplained or schema-only.Scores decline when explanations omit behavioral impact, contain only basic information, or merely list parameters.
  • Examples Balance: Descriptions score 5/5 for examples balance when prose is self-sufficient and examples only supplement it, but 1/5 when only examples appear.Descriptions are labeled Bad if examples replace the description instead of supporting it.
  • Labeling Rules: A description is labeled Good only if all six dimensions score 3 or higher and every component requirement is satisfied; it is Bad if any dimension scores below 3.The output records each component score, a Good-or-Bad label, a one-sentence reason, and weak areas with scores <= 3.

A.2 Prompts used by Tool Description Augmentor

The augmentor uses one prompt to generate realistic, single-call benchmark tasks and another to synthesize execution evidence into concise, factual tool documentation. Both prompts enforce direct, evidence-grounded outputs and explicit formats.

  • A.2.1 Task Generation Prompt.: The task-generation prompt asks an expert benchmark designer to create realistic, diverse tasks testing correct use of a specific MCP tool.Each task is framed as a natural-language question for AI-agent evaluation.
  • A.2.1 Task Generation Prompt.: Each generated task must have one unambiguous answer obtainable through a single tool call and use concise, conversational English.The prompt contrasts direct user-style questions with formal, verbose tool-oriented wording.
  • A.2.1 Task Generation Prompt.: Tasks must specify every required input value and vary realistic parameter values across tasks for the same tool.Examples of required values include dates, tickers, names, and units.
  • A.2.1 Task Generation Prompt.: Difficulty levels range from medium and hard usage to very hard tasks combining constraints, ambiguity resolution, brittle edge cases, or failure boundaries.Even the most demanding tasks must remain solvable with one tool call.
  • A.2.1 Task Generation Prompt.: The task generator must return only a JSON object containing difficulty, question, correct_answer, and rationale fields.The specified schema permits medium, hard, or very hard difficulty labels.
  • A.2.2 Jury LLM Prompt.: The jury prompt converts exact tool arguments and responses or errors into a concise, factual additional_description block that helps agents use the tool correctly.Its input consists of observed executions rather than invented scenarios.
  • A.2.2 Jury LLM Prompt.: The synthesized documentation must contain labeled Examples and Limitations sections, with examples tied to observations and limitations grounded in observed constraints or behaviors.Examples show exact JSON arguments and exact responses or faithful excerpts of at most 20 lines.
  • A.2.2 Jury LLM Prompt.: The jury must prefer edge cases, errors, and boundary values, write in plain direct English, avoid unsupported facts, and return documentation text without a JSON wrapper.It also must not repeat the original tool description verbatim or add a tool-name heading.
Loading 2602.14878v3…