PaperlayerView exact revision on arXiv

Source-linked AI summary

MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use

Zijian Wu, Xiangyan Liu, Xinyuan Zhang, Lingjun Chen, Fanqing Meng, Lingxiao Du, Yiran Zhao, Fanshi Zhang, Yaoqi Ye, Jiawei Wang, Zirui Wang, Jinjie Ni, Yufan Yang, Arvin Xu, Michael Qizhe Shieh

arXiv:2509.24002v1cs.CLcs.AI

TL;DR

Existing MCP benchmarks provide limited coverage of realistic, deep workflows. MCPMark addresses this gap with 127 collaboratively created tasks across curated environments and evaluates models through a standardized tool-calling loop. The best model reaches 52.56% pass@1 but only 33.86% pass^4, while performance varies substantially across environments and repeated runs.

  • Problem

    Existing MCP benchmarks emphasize shallow or read-heavy tasks and limited interaction depth, leaving realistic multi-step MCP workflows insufficiently covered.

  • Method

    MCPMark provides 127 collaboratively created tasks across curated initial states, diverse CRUD workflows, and five MCP environments, evaluated with programmatic verification and a minimal tool-calling agent.

  • Results

    The best-performing model, gpt-5-medium, reaches 52.56% pass@1 and 33.86% pass^4, while performance varies across environments and robustness declines across repeated runs.

  • Takeaways & Limitations

    MCPMark exposes substantial challenges in realistic MCP use, particularly consistent multi-run execution, environment transfer, and efficient tool interaction.

  • Takeaways & Limitations

    The labor-intensive task creation pipeline is difficult to scale, and MCPMark’s steep difficulty limits its utility for evaluating smaller, more efficient models.

Abstract

from arXiv · show

MCP standardizes how LLMs interact with external systems, forming the foundation for general agents. However, existing MCP benchmarks remain narrow in scope: they focus on read-heavy tasks or tasks with limited interaction depth, and fail to capture the complexity and realism of real-world workflows. To address this gap, we propose MCPMark, a benchmark designed to evaluate MCP use in a more realistic and comprehensive manner. It consists of $127$ high-quality tasks collaboratively created by domain experts and AI agents. Each task begins with a curated initial state and includes a programmatic script for automatic verification. These tasks demand richer and more diverse interactions with the environment, involving a broad range of create, read, update, and delete (CRUD) operations. We conduct a comprehensive evaluation of cutting-edge LLMs using a minimal agent framework that operates in a tool-calling loop. Empirical results show that the best-performing model, gpt-5-medium, reaches only $52.56$\% pass@1 and $33.86$\% pass^4, while other widely regarded strong models, including claude-sonnet-4 and o3, fall below $30$\% pass@1 and $15$\% pass^4. On average, LLMs require $16.2$ execution turns and $17.4$ tool calls per task, significantly surpassing those in previous MCP benchmarks and highlighting the stress-testing nature of MCPMark.

1 INTRODUCTION

MCPMark addresses the limited realism of existing MCP benchmarks with a broader benchmark for complex, multi-step interactions across external systems. It combines curated environments, diverse CRUD workflows, collaborative task creation, and automated evaluation.

  • Existing MCP benchmarks emphasize shallow or read-heavy interactions, limiting their coverage of realistic multi-step workflows and capabilities such as reasoning, planning, long-context processing, and tool use.
  • MCPMark evaluates realistic user scenarios across five MCP environments using task instructions, curated initial states, and programmatic verification scripts.
  • The benchmark contains 127 tasks and 38 unique initial states, with broader coverage of create, read, update, and delete operations across diverse workflows.
  • Human experts and language agents collaboratively create and refine tasks, while a minimal standardized tool-calling agent supports evaluation across MCP servers and model providers.
  • Evaluation patterns show consistently low pass^4 performance, substantial variation across MCP environments, and more targeted tool use among successful completions.

2 MCPMARK: STRESS-TESTING COMPREHENSIVE MCP USE

MCPMark is constructed from realistic, curated environments and collaboratively developed tasks spanning multiple MCP services. Its automatic verification and sandboxed execution support reproducible evaluation of diverse multi-step CRUD workflows.

  • Benchmark construction: Each task pairs a natural-language instruction with an automatic verification script and begins from a realistic initial environment state.
  • Task creation pipeline: The human–AI creation pipeline explores environments, increases task complexity, verifies outcomes programmatically, and iterates while preserving realism and automatic verifiability.
  • Quality control: Expert cross-review and month-long community validation check clarity, consistency, realism, and unambiguous evaluation outcomes.
  • Benchmark overview: 127 tasks span five MCP servers and 38 curated initial states, with 20 to 30 tasks per environment and realistic workflows across Notion, GitHub, Filesystem, PostgreSQL, and Playwright.
  • State tracking and management: Sandboxed execution tracks explicit state through initialization, agent execution, automatic verification, and environment reset, supporting safety, reproducibility, and fair comparison.
  • Evaluation agent: MCPMark-Agent exposes MCP tools through a lightweight standardized tool-calling loop without task-specific heuristics or model-specific biases.

3 EXPERIMENTS

The experiments evaluate models with complementary success and consistency metrics on all 127 MCPMark tasks. Results show low overall performance, strong environment differences, sharp robustness losses across runs, and no simple relationship between tool-call effort and accuracy.

  • 3.1 EXPERIMENTAL SETUP: Models are evaluated with pass@1, pass@4, and pass^4, where pass^4 requires all four independent runs to succeed and measures consistency under stochastic generation.
  • 3.2 MAIN RESULTS: 52.56% pass@1 is achieved by gpt-5-medium, while qwen3-coder-plus reaches 24.80% and most proprietary models remain between 15% and 30%.
  • 3.2 MAIN RESULTS: gpt-5-medium reaches 76.19%, 57.50%, and 43.00% pass@1 on PostgreSQL, Filesystem, and Playwright, while most models remain below 25% on Notion and GitHub.
  • 3.2 MAIN RESULTS: Pass^4 falls to 33.86% for gpt-5-medium and 12.60% for claude-sonnet-4, despite pass@4 reaching 68.50% and 44.88%, respectively.
  • 3.2 MAIN RESULTS: Higher turn counts do not guarantee better performance: kimi-k2-instruct often exceeds 30 turns, whereas gpt-5-medium achieves the highest pass@1 with a reasonable turn budget.
  • 3.2 MAIN RESULTS: Higher cost alone does not imply better results, because some expensive runs have lower pass@1 while lower-cost runs can perform better.

4 ANALYSIS

MCPMark analysis examines how reasoning effort, tool-calling behavior, and failure types shape performance across realistic MCP tasks. Results show that additional reasoning helps selectively, while successful execution depends on targeted calls and failures are often subtle or model-specific.

  • 4.1 REASONING MODE AND EFFORT: Higher reasoning effort improves gpt-5 performance unevenly: gpt-5 reaches 52.56% pass@1 at medium effort versus 46.85% at low effort, while gpt-5-mini rises from 8.27% to 30.32%.gpt-5-nano changes only marginally around 4% to 6%.
  • 4.1 REASONING MODE AND EFFORT: Stronger models succeed with fewer, better-targeted calls rather than blind trial-and-error, whereas kimi-k2-instruct often exceeds 30 turns with diminishing success rates.Figure 4 plots successful-run turn distributions, with gray points marking failures and color encoding pass@1.
  • 4.1 REASONING MODE AND EFFORT: Remote services benefit more from reasoning than local services: gpt-5 GitHub performance rises from 27.17% to 50.00%, while PostgreSQL remains between 72% and 76%.Notion rises from 36.61% to 44.64%, whereas Filesystem varies by under 5 percentage points.
  • 4.2 FAILURE BREAKDOWN: Failure analysis separates implicit failures, where verification fails after normal completion, from explicit failures such as overflow, turn limits, abandonment, premature stopping, and malformed calls.This categorization distinguishes subtle capability-driven errors from directly identifiable execution breakdowns.
  • 4.2 FAILURE BREAKDOWN: Implicit failures exceed 50% across models, while explicit failures vary by model, including context overflow for gpt-5-high, looping for kimi-k2-instruct, and malformed calls for gemini-2.5-flash.The observed patterns point to distinct issues in reasoning, context management, planning, and tool use.

5 RELATED WORK

Prior MCP benchmarks broaden tool and domain coverage but generally leave high-fidelity, realistic workflows under-tested. MCPMark addresses this gap with diverse CRUD tasks in containerized environments and programmatic verification.

  • 5 RELATED WORK: MCP connects language models to external tools, APIs, databases, and contextual resources, enabling agents to operate in real environments.This interface motivates benchmarks that assess more than isolated or read-heavy interactions.
  • 5 RELATED WORK: Earlier benchmarks study multi-domain, multi-tool, or large-scale MCP tasks, but primarily emphasize broad coverage or easier execution rather than realistic application workflows.MCP-Universe, LiveMCP-101, and MCP-AgentBench illustrate these different emphases.
  • 5 RELATED WORK: MCPMark uses diverse CRUD operations, containerized settings, programmatic verification, and full environment tracking to support safe, reproducible, fine-grained evaluation.These design choices distinguish it from prior benchmarks’ coverage-focused approaches.

6 DISCUSSION ON LIMITATIONS AND FUTURE DIRECTIONS

The paper identifies future progress needs across models, agent frameworks, server-side tools, and benchmark construction. It also notes that MCPMark is difficult to scale and especially demanding for smaller models.

  • 6 DISCUSSION ON LIMITATIONS AND FUTURE DIRECTIONS: Future agents need stronger reasoning, context efficiency, and execution stability to complete long-horizon MCP tasks reliably.The paper specifically calls for concise tool outputs, better summarization, robust error handling, and self-correction.
  • 6 DISCUSSION ON LIMITATIONS AND FUTURE DIRECTIONS: MCPMark’s task-creation pipeline is labor-intensive and difficult to scale, limiting the production of large training datasets.The authors identify semi-automated generation as one possible direction.
  • 6 DISCUSSION ON LIMITATIONS AND FUTURE DIRECTIONS: The benchmark’s steep task difficulty limits its usefulness for evaluating and guiding smaller, more efficient models.The paper proposes a finer-grained difficulty gradient and shorter execution chains.
  • 6 DISCUSSION ON LIMITATIONS AND FUTURE DIRECTIONS: Future benchmark extensions include ambiguous user intent and a wider variety of MCP servers to test clarification, intent inference, and broader tool use.These additions aim to reflect more complex real-world interactions.

ETHICS STATEMENT

The benchmark construction addresses licensing, task curation, and responsible resource use. Initial states and environments include license information, while tasks are collaboratively annotated by experts and AI agents.

  • ETHICS STATEMENT: Initial states and environments are accompanied by appropriate license information, including self-curated environments checked for licensing compliance.The stated goal is responsible and transparent use of benchmark resources.
  • ETHICS STATEMENT: Benchmark tasks are collaboratively annotated by domain experts and AI agents as part of the curation process.
  • ETHICS STATEMENT: The authors state that these practices maintain ethical standards and comply with relevant regulations.

A DETAILS OF THE TASK CREATION PIPELINE

The task creation pipeline combines environment exploration, human guidance, and agent-generated tasks to produce realistic, verifiable workflows. Tasks are executed end to end and stress-tested with programmatic checks before inclusion.

  • Task creation setup: Task creation starts by selecting a staging or test environment, configuring its MCP server, and defining a seed question to guide exploration.The Playwright example emphasizes examining available tools, interactive elements, navigation, authentication, and validation processes before task generation.
  • Exploration and generation: Agents explore the environment and propose challenging, verifiable tasks that combine discovered workflows with multiple MCP operations rather than simple content reading.The guidance targets realistic browser workflows involving forms, navigation, dynamic content, authentication, and data submission.
  • Human–AI refinement: Human experts iteratively revise task descriptions and verification scripts with agents until subtasks are integrated, sufficiently covered, and objectively checkable.The process explicitly addresses weak verification, shallow exploration, and disconnected subtasks through reviewer feedback and repeated refinement.
  • Execution and verification: Each task is saved with separate description and verification files, then executed through the target MCP and tested for end-to-end executability, clear pass/fail outcomes, and edge-case handling.Difficulty may also be adjusted when a task and checker are valid but insufficiently demanding.
  • MCP environment coverage: The benchmark uses five MCP servers—Filesystem, GitHub, Notion, Playwright, and PostgreSQL—with documented functionality, repositories, and licenses.The servers cover local file operations, project management, page and database interaction, browser automation, and relational database access.

C TASK-LEVEL RESULTS ACROSS MODELS

Task-level evaluation complements aggregate metrics with a matrix showing how often each model succeeds on each task. This exposes performance variation across individual task–model pairs.

  • Task–model success matrix: Figure 6 reports the number of successful runs, from 0 to 4, for every task–model pair in a task–model success matrix.The matrix provides a per-task view that complements aggregate benchmark metrics.

D DETAILED MCP BENCHMARK RESULTS

Detailed MCP results break down benchmark performance and usage by individual service rather than aggregating across all environments. The reported tables cover Filesystem, GitHub, Notion, Playwright, and PostgreSQL.

  • Per-MCP results: Tables 4–8 provide benchmark results separately for Filesystem, GitHub, Notion, Playwright, and PostgreSQL MCPs.These service-level tables correspond to the overall results previously aggregated across all MCPs.
  • Metrics and reporting: Per-MCP tables report success metrics alongside usage statistics, with input and output measured in thousands of tokens and cost reported in USD.Bold and underline identify best and second-best success results, while usage emphasis marks the largest and second-largest values without implying better performance.

E CASE STUDIES BY MCP

The case studies document representative successful and failed trajectories across MCP environments, alongside service-specific turn distributions and resource descriptions. Together, they illustrate how partial completion or incorrect final state can cause verification failure.

  • Successful and failed runs: Successful and failed trajectories contrast complete, verifier-passing workflows with partial or incorrect state changes across Filesystem, GitHub, Notion, Playwright, and PostgreSQL.Examples include correct CSV extraction, CI/CD setup, database retagging, authenticated browsing, and schema construction versus incomplete or incorrect outcomes.
  • Playwright: In the Playwright case, o3 reaches an authenticated state after navigating login, entering credentials, and passing Turnstile, whereas grok-4 fails when Turnstile remains unsolved.The contrast shows that entering credentials alone does not satisfy the task verifier.
  • PostgreSQL: In the PostgreSQL case, grok-code-fast-1 creates and updates tracking tables with indexes and seed rows, while grok-4 leaves schema work and required rows or indexes incomplete.The verifier passes only for the completed database state shown in the successful trajectory.
  • Turn distributions: Figures 23–27 show turn distributions separately for Filesystem, Notion, GitHub, PostgreSQL, and Playwright, complementing the overall turn analysis.The plots are intended to reveal how turn requirements differ by service.
  • Resources and licensing: The appendix documents benchmark resources including Notion templates, GitHub repositories, filesystem components, and PostgreSQL databases with associated licensing information.The listed materials include publicly available, author-created, and open-source resources used in the evaluation environment.
Loading 2509.24002v1…