MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers

Motivation

• Existing API- and MCP-based benchmarks provide limited tool compatibility, domain coverage, workflow length, and testing of fuzzy instructions, multigoal objectives, evidence grounding, and cross-domain orchestration.These limitations hinder evaluation of realistic dependency chains and long-horizon planning.Source: §1 · pp. 1–2

Task generation

• Its LLM-based synthesis pipeline discovers dependency chains from tool I/O signatures, translates them into natural-language tasks, filters for solvability and realism, and creates instruction-minimal fuzzy variants.The fuzzy variants retain each task’s core objective while omitting explicit tool references and execution steps.Source: §1 · p. 2

Evaluation

• MCP-Bench evaluates trajectories with rule-based checks for validity, schema compliance, runtime success, and dependency order, plus rubric-driven LLM-as-a-Judge scoring of completion, tool use, and planning.Prompt shuffling and score averaging are used to improve evaluation stability.Source: Figure 1 · p. 2, §1 · p. 3

Benchmark design

• MCP-Bench connects agents to 28 production-grade MCP servers exposing 250 complementary tools, enabling intra-server dependency chains and cross-server multi-hop workflows.The MCP protocol provides consistent invocation schemas across servers, while the benchmark spans domains including finance, science, and research.Source: §1 · pp. 2–3, Figure 1 · p. 2

Empirical study

• The benchmark evaluates 20 state-of-the-art LLMs on 104 challenging tasks, revealing persistent weaknesses in realistic and complex tool-using scenarios.The study is designed to assess tool schema understanding, multi-hop planning, execution correctness, and strategic reasoning.Source: Figure 1 · p. 2, §1 · p. 3

Benchmarking LLMs

• Benchmarking has progressed from fixed-format evaluations to multi-turn dialogue, simulated tool use, open-ended web navigation, and planning under disruptions, while realistic multi-tool workflows remain insufficiently modeled.MMLU, BIG-bench, and HELM emphasize static evaluation; later benchmarks target reasoning, interaction, tool-based decisions, web navigation, and dynamic planning.Source: §2 · p. 4

Evaluating Tool-using Capability

• Tool-use benchmarks increasingly evaluate planning, execution, tool coordination, dependency reasoning, and verified outcomes, but bespoke toolsets limit realism.Mind2Web and WebArena use hand-crafted interfaces, while τ-Bench, BFCL v3, C3-Bench, and ComplexFuncBench broaden coordination and verification in distinct ways.Source: §2 · p. 4
• MCP-based benchmarks standardize LLM–tool interaction and expose domain-aligned tools, but existing systems remain limited by manual setup or small scale, leaving cross-server real-world evaluation open.MCP-RADAR and MCPWorld require manual setup, whereas MCPEval automates task generation and evaluation across five MCP servers.Source: §2 · p. 4

3.1. Formalization of Tool-using LLM Agent

• MCP-Bench models tool-using agents as POMDPs with planning and invocation actions, tool and state observations, rewards, instructions, and multiple MCP servers.It supports both one-shot global planning and multi-turn planning and observation.Source: §3.1 · p. 4
• The multi-turn workflow plans from prior outputs, executes tools, updates state, compresses long observations, logs trajectories, and produces a final answer after termination.Execution lasts up to 20 rounds or until the agent signals to stop.Source: §3.1 · pp. 4–5

3.2. Important Capabilities for Tool-Using LLM Agents and How MCP-Bench Reflects Them

• MCP-Bench tests schema understanding through strict validation of 250 tools with nested structures, enumerated types, constrained ranges, and required or optional arguments.The benchmark detects subtle invocation violations across schemas ranging from scalar inputs to deeply nested hierarchies.Source: §3.2 · p. 5
• MCP-Bench evaluates tool retrieval under fuzzy instructions by adding 10 distractor servers and over 100 additional tools per task while omitting explicit tool names and steps.Agents must resolve semantic variants, naming inconsistencies, and superficially plausible but irrelevant tools from contextual cues.Source: §3.2 · pp. 5–6
• The benchmark measures long-horizon planning and cross-server orchestration through single- and multi-server tasks lasting up to 20 rounds with sequential, parallel, and multi-objective dependencies.Evaluation covers structural coherence, dependency awareness, parallelism efficiency, and reflective adaptation.Source: §3.2 · p. 6
• MCP-Bench evaluates information grounding by linking execution histories to rubric-based judgments that reward citations of tool outputs and penalize unsupported reasoning.The capability targets factual consistency across calls and traceable evidence for claims.Source: §3.2 · p. 6
• The benchmark assesses real-world adaptability across 28 production-grade MCP servers spanning finance, healthcare, scientific computation, cultural heritage, and other domains.Tasks require interpreting domain-specific semantics, handling diverse tool behaviors, and synthesizing heterogeneous outputs.Source: §3.2 · p. 6

4.1. MCP Server Collection

• MCP-Bench spans 28 representative MCP servers, 250 tools, and eleven functional domains, covering capabilities from scientific computation and finance to geospatial services and analytical utilities.Tool counts range from single-tool servers to BioMCP with 35 tools, Scientific Computing with 26, and Medical Calculator with 22.Source: §4.1 · pp. 6–7

4.2. Task Synthesis

• The benchmark synthesizes 104 tasks across single-server, two-server, and three-server settings, with single-server tasks spanning every included server.The counts are 56 single-server tasks, 30 two-server tasks, and 18 three-server tasks; multi-server combinations are listed in Table 10.Source: §4.2 · p. 7
• Task synthesis discovers tool dependency chains, including natural, scenario-based, cross-server, linear, parallel, and hybrid workflows, then uses them to generate tasks and guide evaluation.For multi-server configurations, cross-server dependencies are emphasized to require coordination across different data sources; dependency analyses also serve as references for the LLM judge.Source: §4.2 · p. 7
• Generated tasks are filtered for solvability and practical utility, retaining only those meeting thresholds of 9.0/10 and 5.0/10, respectively.Tasks below either threshold are discarded, prioritizing benchmark integrity over quantity.Source: §4.2 · p. 7
• Quality-approved tasks are fuzzed into conversational, high-level requests that omit explicit operational details while preserving numerical values and concrete parameters where precise inputs are required.This forces agents to infer tool sequences and execution strategies while keeping domains such as scientific computation and unit conversion mathematically solvable; human inspection additionally checks realism, executability, and dependency-chain reasonability.Source: §4.2 · p. 7

5. Evaluation Method and Metrics

• The framework combines rule-based metrics for name validity, schema adherence, runtime success, and dependency compliance with LLM judging of task completion, tool selection, and planning.Together, these components evaluate both execution correctness and strategic quality.Source: §5 · pp. 7–8

5.1. Rule-based Evaluation

• Rule-based evaluation measures whether agents invoke available tools, provide schema-compliant parameters, and obtain valid results without runtime failures.These metrics quantify grounding in tool availability, understanding of API argument formats, and robust interaction with external systems.Source: §5.1 · p. 8

5.2. LLM-as-a-Judge Evaluation

• An LLM-as-a-Judge framework scores agent performance on task completion quality, tool selection/usage rationale, and planning effectiveness from observable task, solution, and execution-trace evidence, using rubric sub-dimensions normalized to [0, 1].The default judge is o4-mini, with task completion and tool usage dimensions covering fulfillment, grounding, relevance, tool appropriateness, and parameter accuracy.Source: §5.2 · pp. 8–9
• Five independently shuffled rubric prompts are averaged to reduce ordering bias and produce more stable, reliable LLM-judge assessments.Axis and sub-dimension orderings are randomly permuted while rubric semantics remain unchanged; empirical results in Section 6.4 show reduced score variance.Source: §5.2 · p. 9

6.1. Main Results

• gpt-5 (0.749), o3 (0.715), and gpt-oss-120b (0.692) achieve the highest overall MCP-Bench scores, while llama-3-1-8b-instruct reaches 0.428.Table 3 averages results across single-server and multi-server settings; the strongest models combine accurate tool use with robust planning, whereas smaller models show weaker dependency awareness and parallelism.Source: §6.1 · p. 9, Table 3 · p. 10
• Strong models remain stable as server counts increase, whereas weaker models degrade, with llama-3-1-8b-instruct dropping from 0.438 to 0.415 and nova-micro-v1 from 0.520 to 0.471.Performance fluctuations across server counts indicate that sequential dependencies and parallel orchestration create scaling challenges, while gpt-5, o3, and qwen3-235b-a22b-2507 remain competitive above 0.70.Source: §6.1 · pp. 9–10, §6.2 · p. 10

6.2. Agent Performance on Different Capabilities and Insights from MCP-Bench

• Schema compliance and valid tool naming exceed 95% for even mid-scale systems, but planning separates models, with top systems near 0.72 and weaker models rarely above 0.30.Across six axes, gpt-5 reaches dependency awareness 0.76 and parallelism efficiency 0.34, while o3 reaches 0.69 and 0.37, respectively.Source: §6.2 · pp. 10–11

6.3. Number of Rounds and Tool Calls for Different Models Executing Tasks

• llama-3-1-8b-instruct averages 17.3 rounds and over 155 calls per task, whereas stronger models typically achieve comparable or higher success with under 30–40 calls and 6–8 rounds.gpt-5 and gpt-oss-120b use deeper reasoning at 7–9 rounds while controlling call budgets to 48–79 calls.Source: §6.3 · pp. 11–12

6.4. Ablation Studies on LLM Judge Pipeline

• Removing prompt shuffling and score averaging yields CV 16.8%, whereas enabling them lowers it to 15.1%; separately, prompt perturbation raises human agreement from 1.24 to 1.43 out of 2.The consistency evaluation covers 50 benchmark tasks synthesized using WebSearch1 and Time2 MCP servers.Source: §6.4 · pp. 12–13

Conclusion

• MCP-Bench connects agents to 28 production servers and 250 tools, supporting complex multi-hop workflows and cross-domain orchestration.The benchmark uses the Model Context Protocol to evaluate realistic ecosystem-based tool use.Source: §7 · p. 13
• Its automated task synthesis pipeline generates 104 challenging tasks with fuzzy instructions requiring strong agentic capabilities.Tasks are designed to test agents beyond explicit tool-name matching.Source: §7 · p. 13
• Evaluation combining rule-based checks and LLM Judge scoring reveals that even state-of-the-art models struggle with dependency compliance, noisy-environment tool selection, and long-horizon planning.These results expose weaknesses across multiple agentic capabilities.Source: §7 · p. 13

A.1 MCP Servers

• Section A.1 details the MCP servers used in the study.Source: A. Appendix · p. 18

A.2–A.4 Prompts

• Sections A.2–A.4 present prompts for task execution, task synthesis, and evaluation, respectively.Source: A. Appendix · p. 18

A.5–A.6 Schemas and Tasks

• Sections A.5–A.6 show examples of tool input schemas and provide additional task details.Source: A. Appendix · p. 18

Server and tool catalog

• Table 8 provides detailed descriptions of the involved MCP servers and their associated tools.The catalog identifies each server, repository, tool count, functional description, and tool-level capabilities.Source: Table 8 · p. 18, §A.1 · p. 18

Biomedical research

• BioMCP combines biomedical literature, clinical-trial, genetic-variant, and AlphaGenome analysis tools with structured research-planning support.Its tools include sequential thinking, article and trial retrieval, variant analysis, and AlphaGenome predictions.Source: §A.1 · p. 19

Academic discovery

• The catalog includes academic discovery services such as Call for Papers and Paper Search for finding conferences, scholarly literature, PDFs, and full-text content.Paper Search spans arXiv, PubMed, bioRxiv, medRxiv, Google Scholar, IACR, and Semantic Scholar, with download and reading tools.Source: §A.1 · pp. 19, 27

Market, software, and real-world data

• Other servers provide specialized access to live market, software, location, and gaming data, including DEX Paprika, Context7, Google Maps, and Game Trends.Several workflows impose ordered usage, such as retrieving supported networks before querying DEX data or resolving a library ID before fetching documentation.Source: §A.1 · pp. 20–21

Specialized domain services

• The part also documents domain-specific servers for medical calculation, OSINT reconnaissance, NASA data, NixOS configuration, cryptocurrency exchange data, and scientific computing.These tools support clinical risk and dosing calculations, security reconnaissance, astronomy and space-weather queries, package and configuration search, market analysis, and tensor-based computation.Source: §A.1 · pp. 22, 23, 25, 26, 28

Mathematical Computing

• The mathematical computing server supports matrix storage and operations, decompositions, basis transformations, vector algebra, symbolic calculus, and function or vector-field visualization.Its tools include inversion, determinant, rank, eigenvalue computation, QR and SVD decompositions, gradients, curl, divergence, Laplacians, directional derivatives, and plotting.Source: §A.1 · p. 28

Time

• The time service provides current-time retrieval by IANA timezone and conversion between specified source and target time zones.Source: §A.1 · p. 28

Unit Converter

• The unit-conversion server covers temperature, angle, length, energy, force, pressure, power, speed, area, mass, volume, data, density, time, weight, and batch conversions.It includes dedicated conversion tools for common measurement categories and a batch-processing function.Source: §A.1 · p. 29

Weather Data

• The weather server supplies current conditions, 1–10-day forecasts, location search, and live temperature retrieval for global locations.Weather outputs include temperature, conditions, humidity, wind, and detailed meteorological predictions.Source: §A.1 · p. 29

• The section provides the task execution agent’s detailed prompt for MCP-Bench.Source: §A.2 · p. 31

• The agent first assesses whether the original task is complete and, if not, whether another round would provide significant value.Source: Strategic Planning Prompt · p. 31
• For a continuing round, the prompt requires planning all tool calls in parallel, prioritizing calls that do not require dependencies.Source: Strategic Planning Prompt · p. 31
• Planning builds on previous-round results, focuses on independent tasks supported by current information, and avoids redundant successful calls unless needed.Source: Strategic Planning Prompt · p. 31

• The synthesizer must produce a final, comprehensive, well-structured answer based on the original task and information gathered from multiple servers.The answer must directly address the user’s request rather than merely restating intermediate execution results.Source: Final Solution Generation Prompt · p. 32
• After {total_executions} tool calls across multiple MCP servers, the process supplies accumulated information and results for final synthesis.The prompt positions the completed multi-round execution and its accumulated outputs as the basis for answer generation.Source: Final Solution Generation Prompt · p. 32
• The response should consolidate key findings, highlight contributions from different tools and servers, and present them in a clear, logical structure.These guidelines require showing how the different server capabilities were combined while addressing all aspects of the original task.Source: Final Solution Generation Prompt · p. 32
• Where appropriate, the synthesis should conclude with clear, actionable conclusions.Source: Final Solution Generation Prompt · p. 32

A.3. Details of the Used Prompt for Task Synthesis

• The section provides the detailed task-synthesis prompt used in MCP-Bench.Source: §A.3 · p. 33

STEP 1: ANALYZE AND CREATE TOOL DEPENDENCIES

• The generation prompt requires analyzing inherent and scenario-based tool dependencies, including data flow, decision points, sequential or parallel execution, iterative re-analysis, and cross-server interactions.Dependency analysis records tool chains, critical decisions, parallel versus sequential requirements, and fallback or validation relationships across servers.Source: STEP 1: ANALYZE AND CREATE TOOL… · p. 33

STEP 2: DESIGN ONE COMPLEX TASK

• Each generated task must maximize complexity through multiple tool calls, deep dependency chains, intermediate-result decisions, iterative refinement, cross-validation, transformation, and conditional workflows.The task must use available servers, follow a specific tool sequence, define expected outputs, remain executable without clarification, and provide realistic business or research value.Source: STEP 2: DESIGN ONE COMPLEX TASK · pp. 33–34
• Generated tasks are self-contained and executable without external systems, requiring concrete values, named entities, exact thresholds, relative dates, and all inputs needed for searches, calculations, comparisons, or optimization.The prompt forbids vague references, URLs, files, databases, and unspecified parameters while requiring data to come from tools or explicit task values.Source: STEP 2: DESIGN ONE COMPLEX TASK · p. 34

Task Quality Assessment Prompt

• Task quality is assessed separately by solvability and utility on 1–10 scales, considering tool and data sufficiency, measurable success criteria, practical value, actionability, and whether the complexity tests meaningful capabilities.Solvability asks whether the goal can be completed with the available tools and data, whereas utility asks whether it addresses a valuable business or research need.Source: Task Quality Assessment Prompt · p. 35

Task Description Fuzzing Prompt

• Detailed tasks are fuzzed into natural conversational requests that conceal task structure, tool names, server names, and formal instructions while retaining critical information and evidence requirements.The fuzzy prompt uses contextual stories, uncertainty, personal motivation, contractions, approximate language where appropriate, and relative dates rather than explicit procedural descriptions.Source: Task Description Fuzzing Prompt · p. 36, CRITICAL: CREATE A GENUINELY NATURAL… · pp. 36–37

LLM Judge Prompt

• The LLM judge evaluates multi-server execution objectively from the task, concrete reference, dependency analysis, solution, execution summary, tool usage, and available tools, while ignoring fluency, formatting, and narrative style.The judge must justify scores with evidence, recognize that the agent saw only the fuzzy task, and assess completion against the concrete reference.Source: §A.4 · p. 38, LLM Judge Prompt · p. 38

2. Grounding

• The rubric scores task fulfillment, grounding, tool appropriateness, parameter accuracy, dependency awareness, and parallelism or efficiency according to the percentage of requirements, claims, calls, parameters, or dependencies executed perfectly.Scores are normalized by opportunity count and task complexity, with higher scores requiring optimal selection, complete parameters, zero redundancy, proper recovery, efficient parallelism, and concise output.Source: LLM Judge Prompt · p. 38, §2 · pp. 38–39

2. Parallelism and Efficiency

• The scoring procedure applies strict defect-rate bands, evaluates proportional success rather than absolute steps, defaults most executions to moderate scores, and treats missing required JSON as a failed requirement.A portion counts as perfectly executed only when all required conditions are met, and any imperfection lowers the perfect-execution percentage.Source: §2 · pp. 39–41

Input Schemas and Task Construction

• The examples include structured input schemas for patient sex and blood pressure, eGFR, tensor creation, and basis changes.The blood-pressure schema restricts sex to male or female, systolic pressure to integers from 50 to 250 mmHg, and diastolic pressure to integers from 30 to 150 mmHg.Source: Input Schema Example 1: Blood Pressure… · p. 43, Input Schema Example 2:… · p. 44, Input Schema Example 3: Tensor… · p. 44

A.6. Details of the Tasks

• MCP-Bench demonstrates more task examples in Table 9 and lists the server combinations used to construct those tasks in Table 10.Source: §A.6 · p. 47, Table 9 · p. 47, Table 10 · p. 52
• A news-classification task combines Hugging Face, Paper Search, and Wikipedia to identify a four-category model, compare F1 scores with recent papers, explain improvements, and clarify microaveraged F1.It requests concrete model identifiers, reported metrics, recent-paper comparisons, architectural or training changes, and verifiable references.Source: §A.6 · p. 48
• A five-day camping task combines Google Maps and National Parks to select a park near Denver, optimize campsite elevations, retrieve alerts and events, and produce restaurant-linked daily directions.The requested itinerary includes campsite, visitor-center, restaurant, rating, distance, elevation, and schedule information.Source: §A.6 · p. 48
• A reproducible Flask-environment task combines NixOS and Context7 to retrieve pinned package versions, platform configuration options, channel information, documentation, and a possible Poetry flake.The requested setup covers Python 3.10, Flask, Redis, Docker, Nix channel 25.05, Home Manager, and nix-darwin.Source: §A.6 · p. 49
• A museum-and-Wikipedia task asks the agent to identify New Kingdom Egyptian seating at the Metropolitan Museum, compare materials with Wikipedia facts, and synthesize object and contextual evidence.The workflow requires department discovery, object IDs and images, dates, materials, summaries, key facts, construction details, related topics, and unmatched materials.Source: §A.6 · p. 49
• A portfolio-analysis task uses Scientific Computing and Math MCP tools to assess covariance-matrix conditioning, choose an inverse or pseudoinverse, calculate weights, and verify projections and normalization.The requested outputs include determinant, eigenvalues and eigenvectors, condition number, inverse choice, matrix, portfolio weights, projection, and weight sum.Source: §A.6 · p. 50
• A clinical-and-nutrition task combines Medical Calculator, FruityVice, and BioMCP tools to compute cardiometabolic scores, laboratory corrections, treatment-related quantities, and fruit nutrition facts.It requests BMI, BSA, eGFR, creatinine clearance, cardiovascular risk, statin guidance, CHA2DS2-VASc, corrected sodium and calcium, fluids, steroid conversion, and apple and banana nutrition.Source: §A.6 · p. 51