PaperlayerView exact revision on arXiv

Source-linked AI summary

MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers

Zhenting Wang, Qi Chang, Hemani Patel, Shashank Biju, Cheng-En Wu, Quan Liu, Aolin Ding, Alireza Rezazadeh, Ankit Shah, Yujia Bao, Eugene Siow

arXiv:2508.20453v1cs.CL

TL;DR

Existing tool-use benchmarks provide limited evidence about agents operating across realistic, complementary tools under fuzzy instructions and long-horizon objectives. MCP-Bench addresses this gap with a large-scale MCP-based benchmark and structured evaluation framework. Experiments across advanced models reveal persistent weaknesses in complex tool-using scenarios, especially dependency compliance, tool selection, and long-horizon planning.

  • Problem

    Existing benchmarks inadequately evaluate fuzzy tool retrieval, multi-hop planning, evidence grounding, and cross-domain orchestration because they emphasize explicit tools, shallow workflows, or isolated APIs.

  • Method

    MCP-Bench connects agents to 28 MCP servers exposing 250 tools, synthesizes 104 fuzzy multi-hop tasks, and evaluates trajectories with rule-based checks plus rubric-driven LLM judging.

  • Results

    Advanced models show persistent weaknesses in dependency chain compliance, noisy-environment tool selection, and long-horizon planning, while stronger systems remain more stable as server complexity increases.

  • Takeaways & Limitations

    MCP-Bench provides a standardized platform for evaluating agentic reasoning and tool use in realistic ecosystem-based workflows.

Abstract

from arXiv · show

We introduce MCP-Bench, a benchmark for evaluating large language models (LLMs) on realistic, multi-step tasks that demand tool use, cross-tool coordination, precise parameter control, and planning/reasoning for solving tasks. Built on the Model Context Protocol (MCP), MCP-Bench connects LLMs to 28 representative live MCP servers spanning 250 tools across domains such as finance, traveling, scientific computing, and academic search. Unlike prior API-based benchmarks, each MCP server provides a set of complementary tools designed to work together, enabling the construction of authentic, multi-step tasks with rich input-output coupling. Tasks in MCP-Bench test agents' ability to retrieve relevant tools from fuzzy instructions without explicit tool names, plan multi-hop execution trajectories for complex objectives, ground responses in intermediate tool outputs, and orchestrate cross-domain workflows - capabilities not adequately evaluated by existing benchmarks that rely on explicit tool specifications, shallow few-step workflows, and isolated domain operations. We propose a multi-faceted evaluation framework covering tool-level schema understanding and usage, trajectory-level planning, and task completion. Experiments on 20 advanced LLMs reveal persistent challenges in MCP-Bench. Code and data: https://github.com/Accenture/mcp-bench.

1. Introduction

MCP-Bench addresses limitations in existing tool-use benchmarks by evaluating realistic, multi-step workflows across complementary MCP tools. It combines fuzzy task generation with trajectory-level and execution-level evaluation.

  • Existing benchmarks often use isolated APIs, shallow tool-call sequences, and explicit execution steps, limiting realistic tool-use evaluation.Compatible APIs can support cleaner compositions, but coverage remains limited across domains and tools.
  • The benchmark evaluates tool schema understanding, multi-hop planning, and adaptability in realistic tool-use scenarios.Its design targets capabilities underrepresented by benchmarks built around explicit tools and shallow workflows.
  • MCP-Bench connects agents to 28 MCP servers exposing 250 structured tools for intra-server dependency chains and cross-server workflows.The servers provide complementary tools with consistent invocation schemas across domains.
  • Tasks are synthesized from tool input-output dependencies, filtered for solvability and realism, and rewritten as fuzzy instructions without explicit tool references.This requires agents to infer appropriate tools and execution strategies from high-level objectives.
  • Execution trajectories are evaluated with rule-based checks and rubric-driven LLM-as-a-Judge scoring for validity, compliance, completion, tool use, and planning.Prompt shuffling and score averaging are used to improve evaluation stability.

2. Related Work

Related benchmarks have progressed from static reasoning tests toward interactive tool-use evaluation. However, existing approaches still rely on fixed or hand-crafted toolsets and differ in how broadly they test coordination.

  • MMLU and BIG-bench primarily evaluate single-turn or fixed-format knowledge and reasoning, while HELM broadens static evaluation across multiple quality dimensions.These benchmarks are not centered on dynamic execution through external tools.
  • Mind2Web and WebArena evaluate think-to-act planning with browser environments, but depend on hand-crafted toolsets.Their settings provide interactive evaluation without the broad, ecosystem-based tool spaces targeted by MCP-Bench.
  • Tool-use benchmarks increasingly target reasoning, planning, and execution across tool interfaces, with later work pursuing broader tool coordination.The related-work trajectory motivates more realistic multi-tool evaluation.

3. MCP-Bench Formalization and Design Principles

MCP-Bench formalizes tool-using agents as multi-server POMDPs and evaluates capabilities needed for realistic, fuzzy, long-horizon workflows. Its design covers schema compliance, retrieval, orchestration, grounding, and adaptability.

  • 3.1. Formalization of Tool-using LLM Agent: Each task is modeled as a POMDP tuple containing global states, planning and tool actions, observations, transitions, rewards, instructions, and available MCP servers.Servers expose tool sets whose union defines the complete action space.
  • 3.1. Formalization of Tool-using LLM Agent: Agents support one-shot global planning and multi-turn planning with observations from tool executions and internal state.The formulation explicitly accommodates multiple external servers and tools.
  • 3.1. Formalization of Tool-using LLM Agent: In each round, planning conditions on prior outputs, execution performs the selected tools, and the agent updates its internal state for up to 20 rounds or until stopping.The workflow logs plans and observations before producing a final answer from the trajectory.
  • 3.2. Important Capabilities for Tool-Using LLM Agents and How MCP-Bench Reflects Them: MCP-Bench enforces precise schema understanding across nested structures, enumerated types, constrained ranges, and required or optional arguments.Strict validation detects subtle invocation violations across 250 tools.
  • 3.2. Important Capabilities for Tool-Using LLM Agents and How MCP-Bench Reflects Them: Fuzzy retrieval is tested with 10 distractor servers adding 100+ tools per task, while task descriptions omit explicit tool names and execution steps.Agents must infer appropriate tools from contextual cues despite heterogeneous and inconsistent tool spaces.
  • 3.2. Important Capabilities for Tool-Using LLM Agents and How MCP-Bench Reflects Them: Long-horizon orchestration spans single- and multi-server tasks, requiring agents to manage sequential and parallel dependencies across up to 20 execution rounds.The setting includes heterogeneous outputs and potentially simultaneous goals.
  • 3.2. Important Capabilities for Tool-Using LLM Agents and How MCP-Bench Reflects Them: Grounding evaluation rewards answers that cite actual tool outputs and penalizes unsupported reasoning based on execution history.This directly tests factual consistency across tool calls.
  • 3.2. Important Capabilities for Tool-Using LLM Agents and How MCP-Bench Reflects Them: Real-world adaptability requires interpreting domain semantics, handling diverse tool behaviors, and synthesizing heterogeneous outputs across 28 production-grade servers.Covered domains include finance, healthcare, scientific computation, and cultural heritage.

4. MCP-Bench Construction

MCP-Bench constructs a diverse tool ecosystem and converts its natural input-output dependencies into realistic benchmark tasks. A three-stage pipeline discovers dependencies, filters quality, and fuzzes task descriptions.

  • The benchmark covers 28 MCP servers across eleven functional domains, providing 250 tools with widely varying server sizes.Domains range from media and research to finance, science, travel, health, weather, time, and divination.
  • Task construction begins by discovering dependency chains where one tool’s outputs naturally flow into another tool’s inputs.The pipeline uses both inherent tool relationships and scenario-based dependencies, emphasizing cross-server links in multi-server configurations.
  • Generated tasks undergo two-dimensional filtering for solvability and practical utility before entering the benchmark.Tasks below 9.0/10 solvability or 5.0/10 utility are discarded, reducing quantity to preserve quality.
  • Passed tasks are transformed into fuzzy descriptions that state high-level goals without explicit operational details.Agents must infer tool sequences and execution strategies from the available dependency structures.

5. Evaluation Method and Metrics

MCP-Bench combines rule-based execution metrics with LLM-as-a-Judge scoring to evaluate both correctness and strategic quality. The framework measures tool validity, schema adherence, runtime success, task completion, tool usage, and planning effectiveness, while shuffling rubric order and averaging scores to mitigate judge-order sensitivity.

  • Rule-based Evaluation: The rule-based evaluation measures name validity, schema adherence, runtime success, and dependency compliance from execution traces.These dimensions assess whether agents use available tools, provide valid parameters, execute successfully, and respect tool dependencies.
  • Rule-based Evaluation: Tool Name Validity Rate measures whether each invocation selects a tool from the allowed set.The metric penalizes hallucinated or invalid tool references and reflects grounding in tool availability.
  • Rule-based Evaluation: Schema Compliance Rate measures whether invocations provide parameters matching each tool’s expected input schema.This evaluates understanding of API argument formats and avoidance of malformed requests.
  • Rule-based Evaluation: Execution Success Rate measures the proportion of invocations that return valid results without runtime errors.A high rate indicates robust interaction with external systems and proper error handling.
  • LLM-as-a-Judge Evaluation: The LLM judge scores task completion quality, tool usage quality, and planning effectiveness using observable task, solution, and execution-trace evidence.The axes cover fulfillment and grounding, tool appropriateness and parameter accuracy, and dependency awareness, parallelism, and efficiency.
  • LLM-as-a-Judge Evaluation: Prompt shuffling randomly permutes evaluation axes and sub-dimensions before score averaging to reduce sensitivity to rubric ordering.The semantic content and phrasing of the rubrics remain unchanged during shuffling.

6. Benchmark Results

MCP-Bench results show that execution fidelity is broadly strong, while planning, dependency awareness, and robustness across multi-server workflows remain key differentiators. Stronger models sustain higher performance and use more controlled interaction budgets than weaker systems.

  • 20 LLMs were evaluated on MCP-Bench across single-server and multi-server settings.
  • 0.438 to 0.415 and 0.520 to 0.471 were the overall-score declines for llama-3-1-8b-instruct and nova-micro-v1 as settings expanded from single-server to multi-server.The main declines involved dependency awareness and parallelism, and performance fluctuated across server counts.
  • 0.75 was gpt-5's approximate overall score across both settings, while o3 and qwen3-235b-a22b-2507 remained above 0.70.
  • 0.76 was gpt-5's dependency-awareness score, compared with 0.69 for o3 and 0.54 for qwen3-235b-a22b-2507.Smaller models rarely exceeded 0.30 on dependency awareness or parallelism efficiency.
  • Above 95% accuracy was achieved by even mid-scale systems on schema compliance and valid tool naming.The results identify basic execution fidelity as less limiting than higher-order planning and coordination.
  • 17.3 rounds and over 155 calls per task were used on average by llama-3-1-8b-instruct, while stronger models typically stayed under 30–40 calls and 6–8 rounds.gpt-5 and gpt-oss-120b used 7–9 rounds with 48–79 calls, balancing deeper reasoning with controlled call budgets.
  • 16.8% to 15.1% was the coefficient-of-variation reduction from removing to enabling prompt shuffling and score averaging.Human agreement also increased from 1.24 to 1.43 out of 2 with prompt perturbation.

7. Conclusion

MCP-Bench evaluates realistic ecosystem-based tool use and finds that even state-of-the-art models struggle with dependency compliance, tool selection, and long-horizon planning.

  • MCP-Bench connects agents to 28 production servers exposing 250 tools for complex multi-hop and cross-domain workflows.Its task synthesis pipeline generates 104 tasks with fuzzy instructions.
  • The benchmark combines rule-based execution checks with LLM Judge scoring to assess tool use and agentic capabilities.The evaluation targets dependency chains, tool selection under noisy conditions, and long-horizon planning.
  • Even state-of-the-art models struggle with dependency chain compliance, noisy-environment tool selection, and long-horizon planning.

A. Appendix

The appendix documents the benchmark’s MCP servers, execution and evaluation prompts, tool schemas, and task examples.

  • The appendix details the MCP servers used in the benchmark.
  • It presents prompts for task execution, task synthesis, and evaluation.
  • It includes tool input-schema examples and additional task details.

A.1. Details of Used MCP Servers

The benchmark uses a broad collection of MCP servers spanning programming, finance, gaming, maps, biomedical research, academic events, automotive pricing, and other services.

  • The appendix lists server descriptions and associated tools for the benchmark’s MCP ecosystem.Table 8 provides detailed descriptions for the involved servers and tools.
  • BioMCP supports biomedical literature, clinical-trial, genetic-variant, and AlphaGenome-related research workflows.
  • Context7 provides library identification and current documentation retrieval through two tools.
  • DEX Paprika provides blockchain network, decentralized-exchange, liquidity-pool, token, historical-price, and transaction analytics.
  • Other listed servers provide fruit nutrition, gaming trends, Google Maps, icon search, Hugging Face resources, OSINT, conference discovery, and car-price data.

A.2. Details of the Used Prompt for the Task Execution Agent

The appendix documents the prompt used by the task execution agent in MCP-Bench.

  • The appendix presents the detailed prompt used for task execution.
  • The prompt is associated with the task execution agent in MCP-Bench.
  • The prompt documentation appears in the section covering task execution-agent instructions.

Strategic Planning Prompt

The strategic planning prompt guides a multi-tool agent through round-based completion assessment and parallel execution planning. It emphasizes using current information, avoiding redundancy, and executing independent calls simultaneously.

  • The role instruction frames the agent as responsible for strategic decision-making and tool planning across multiple rounds.
  • The prompt supplies the task, current round, and available tools across servers as the planning context.
  • The agent first assesses whether the original task is complete before deciding whether another execution round is worthwhile.
  • Each continuing round should contain all planned tool calls that can execute in parallel without dependencies.
  • The prompt directs the agent to avoid redundant calls, build on previous results, and focus on independent tasks supported by current information.
  • The prompt requires a JSON response format for the planning output.

Final Solution Generation Prompt

The final solution generation prompt asks an expert synthesizer to combine information from completed multi-round tool execution into a comprehensive answer. It emphasizes integrating server contributions, addressing the original task, and presenting clear conclusions.

  • The prompt identifies the synthesizer as an expert solution synthesizer for multi-tool agent execution.
  • The synthesizer receives accumulated information and the total number of tool calls from multiple MCP servers.
  • The final answer must directly address the original task using information gathered across all execution rounds.
  • The requested presentation is clear, organized, logical, and structured around all aspects of the original task.
  • The prompt requires consolidating key findings and explaining how different tools and servers contributed to the solution.
  • The synthesizer is instructed to provide actionable conclusions where appropriate.

A.3. Details of the Used Prompt for Task Synthesis

This appendix section introduces the detailed prompt used for task synthesis in MCP-Bench.

  • The section presents the detailed prompt used to synthesize tasks in MCP-Bench.

Task Generation Prompt

The task-generation pipeline designs complex, executable, realistic tasks by creating tool dependencies, assessing solvability and utility, and converting detailed specifications into natural user requests. It hides implementation structure while preserving necessary data and reasoning challenges.

  • Task Generation Prompt: Task Generation Prompt: The task designer analyzes available tools and creates meaningful dependencies for the task scenario.
  • STEP 1: ANALYZE AND CREATE TOOL DEPENDENCIES: STEP 1: ANALYZE AND CREATE TOOL DEPENDENCIES: Dependencies may be inherent, scenario-based, parallel, iterative, cross-validating, or cross-server.
  • STEP 2: DESIGN ONE COMPLEX TASK: STEP 2: DESIGN ONE COMPLEX TASK: The pipeline creates one task with maximum complexity, broad server coverage, and deep dependencies when suitable.
  • STEP 2: DESIGN ONE COMPLEX TASK: STEP 2: DESIGN ONE COMPLEX TASK: Tasks must be self-contained, executable without external dependencies, and based on data supplied by tools or concrete task values.
  • STEP 2: DESIGN ONE COMPLEX TASK: STEP 2: DESIGN ONE COMPLEX TASK: Each task must require multiple tools in sequence, include decision points, define an output format, and be immediately solvable.
  • Task Quality Assessment Prompt: Task Quality Assessment Prompt: Generated tasks are evaluated separately for solvability and utility using criteria ranging from 1 to 10.
  • Task Description Fuzzing Prompt: Task Description Fuzzing Prompt: Detailed tasks are rewritten as natural conversational requests that test reasoning without naming tools, servers, or implementation details.
  • CRITICAL: CREATE A GENUINELY NATURAL REQUEST: CRITICAL: CREATE A GENUINELY NATURAL REQUEST: The fuzzing preserves critical values while using uncertainty, personal context, contractions, and scenario-based examples.
Loading 2508.20453v1…