MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers

1 Introduction

• Existing benchmarks inadequately assess LLMs on real-world MCP workflows, focusing on isolated capabilities or relying on derivative datasets and GUI-based workflows.The introduction identifies limited real-application coverage and data-leakage concerns in prior approaches.Source: §1 · p. 1
• MCP-Universe comprises 231 tasks across six domains and 11 MCP servers, capturing realistic tool use, long-horizon calls, long contexts, scattered evidence, and large tool spaces.The servers connect to actual data sources and environments; domains include navigation, repository management, finance, 3D design, browser automation, and web searching.Source: §1 · pp. 1–2
• Its execution-based framework combines format, static, and dynamic evaluators, with dynamic evaluation retrieving real-time ground truth for time-sensitive tasks.The framework is extensible to new agents and MCP servers and includes a user interface.Source: §1 · p. 2
• GPT-5 achieves a 43.72% success rate, Grok-4 33.33%, and Claude-4.0-Sonnet 29.44%, revealing limitations in real-world MCP environments.The experiments span all six core domains and 11 MCP servers.Source: §1 · p. 2
• LLM agents face long-context degradation and context overflow as interaction steps grow, unfamiliar-tool usage challenges, and significant cross-domain success-rate variation.These findings point to domain-specific optimization needs.Source: §1 · p. 2
• Enterprise-level agents like Cursor cannot outperform standard ReAct frameworks on MCP-Universe.Source: §1 · p. 2

Model Context Protocol

• MCP, introduced by Anthropic in late 2024, standardizes AI connections to external data and tools through JSON-RPC 2.0 over STDIO and SSE, using hosts, clients, and servers.Servers expose tools, prompts, and resources through standardized interfaces.Source: §2 · p. 3

LLMs as Agents

• LLM agents have advanced from text-generation chatbots to systems capable of autonomous planning, reasoning, and tool use, supported by patterns such as ReAct, Reflection, and Plan-and-Solve.Multimodal LLMs have also enabled GUI-based computer-use agents, including OpenAI’s CUA, Anthropic’s Computer-Use, and ByteDance’s UI-Taris.Source: §2 · p. 3

Evaluation of Agents

• Agent benchmarks cover web navigation, GUI interaction, software engineering, and function or tool calling, including WebArena, OSWorld, SWE-bench, and ToolBench.These benchmark families assess interaction with realistic web applications, computer interfaces, software development, and external tools or APIs.Source: §2 · p. 3
• MCP-Universe addresses limitations of MCPWorld, MCP-RADAR, MCPEval, and LiveMCPBench by evaluating authentic MCP servers with time-sensitive scenarios and execution-based task-completion metrics.MCPWorld relies heavily on GUIs and omits time-varying tasks; MCP-RADAR uses less real-world tasks and static ground truth; MCPEval and LiveMCPBench use LLM judges, which are unsuitable for real-time knowledge and can show style bias.Source: §2 · p. 3, Table 1 · p. 3

3.2 Evaluation Framework

• The framework dynamically configures agent-model combinations, MCP servers, and evaluators, then assesses agent-server interactions with objective execution-based pass/fail checks.It captures interaction logs and uses domain-specific validation, including stop-type checks for Google Maps and branch checks for GitHub.Source: Figure 2 · p. 4, §3.2 · pp. 4–5

3.1 Overview

• The benchmark reports success through automated checks after agents select and sequence available MCP tools to achieve a specified goal from an initial context.Tasks require reasoning over partial information, adapting to diverse interfaces, and handling ambiguity or tool-response failures; performance is aggregated across tasks.Source: §3.1 · p. 4

3.3 Real-World MCP Servers

• MCP-Universe uses 11 real-world MCP servers offering 133 tools across six core application domains, grounding evaluation in practical rather than simulated environments.Additional servers such as Notion, Weather, Date, and Calculator support tasks and increase complexity.Source: §3.3 · pp. 5–6

3.4 Tasks and Evaluators

• Authors manually designed challenging tasks across six domains, replacing tasks that LLMs could solve without MCP or reliably solve within five retries.Each domain contains four or five task types, and authors cross-check tasks for feasibility, ambiguity, and correctness.Source: §3.4 · p. 6
• Evaluators comprise format checks, manually specified static-answer checks, and dynamic checks that retrieve real-time ground truth for time-sensitive tasks.The authors argue execution-based evaluation avoids static-knowledge, style-bias, and hallucination limitations of LLM judges; evaluators are cross-checked for feasibility, ambiguity, and correctness.Source: §3.4 · p. 6

4.1 Setup

• Experiments compare top-ranked proprietary and open-source LLMs using ReAct, except GPT-OSS, which uses OpenAI Agent SDK because it cannot follow the ReAct prompt.The tested open-source models have at least 120B parameters; reported measures include success rate, average evaluator score, and average steps for successful tasks.Source: §4.1 · pp. 6–7, Table 3 · p. 7

4.2 Frontier Models Performance

• GPT-5 leads overall at 43.72% success, ahead of Grok-4 at 33.33% and Claude-4.0-Sonnet at 29.44%.GPT-5 also leads in Financial Analysis (67.50%), 3D Designing (52.63%), and Web Searching (45.45%). GLM-4.5 is the best open-source model at 24.68% overall.Source: Table 3 · p. 7, §4.2 · p. 7
• Non-reasoning models exceed 90% success with format evaluators, while GPT-5 leads static evaluators at 61.92% and dynamic evaluators at 65.96%.The passage reports that most models achieve 40–60% on content evaluators, indicating that content generation, rather than format compliance, is the primary source of failure.Source: Table 4 · p. 7, §4.2 · p. 8

4.3 Long Context Challenges

• Context grows rapidly with interaction steps, especially in Location Navigation, Browser Automation, and Financial Analysis; summarization improves some Location Navigation results but has no effect or hurts elsewhere.MCP outputs can include detailed place information, full webpage HTML, or extensive stock histories; the tested summarization approach yields mixed results and is insufficient as a general solution.Source: Figure 4 · p. 8, §4.3 · pp. 8–9

4.4 Unknown Tools Challenges

• Exploration before ReAct-style task execution raises GPT-4.1 Browser Automation success to 30.77% (+7.69 percentage points) and Claude-4.0-Sonnet Financial Analysis to 62.50% (+7.50 percentage points).The gains are not universal: Claude-4.0-Sonnet declines in Browser Automation, and GPT-4.1 shows no Financial Analysis improvement.Source: Figure 5 · p. 9, §4.4 · p. 9

4.5 More MCP Servers Connected

• Connecting 7 MCP servers comprising 94 tools reduced success, including Claude-4.0-Sonnet Location Navigation from 22.22% to 11.11% and GPT-4.1 Browser Automation from 23.08% to 15.38%.GPT-4.1 Financial Analysis success also fell from 40.00% to 35.00%.Source: Figure 6 · p. 9, §4.5 · p. 9

4.6 Enterprise-Level Agent Framework Comparison

• o3 + OpenAI Agent SDK reaches 31.60% overall versus o3 + ReAct’s 26.41%, while Claude-4.0-Sonnet ReAct scores 29.44% versus Cursor Agent’s 26.41%.Cursor Agent scores higher in Browser Automation (43.59% vs. 38.46%) but lower in Web Searching (7.27% vs. 21.82%). The OpenAI Agent SDK configuration has the highest overall performance among tested agent-backbone combinations.Source: Table 5 · p. 10, §4.6 · p. 10

5 Conclusion

• MCP-Universe is the first comprehensive benchmark designed to rigorously evaluate LLMs in real-world MCP environments using authentic data and execution-based evaluators.It exposes capability gaps across long-context handling, tool unfamiliarity, and cross-domain performance.Source: §5 · p. 10
• Even top-ranked models and enterprise-level agents struggle with MCP-driven task complexity, underscoring the need for advances in model design and agent integration.Source: §5 · p. 10
• MCP-Universe’s extensible framework and user-friendly interface provide a testbed for accelerating progress in robust, real-world LLM applications.Source: §5 · p. 10

A MCP Servers

• Most benchmark servers are official; the others are based on official APIs, and Table 6 lists the server names and links.The list is intended to help users use the benchmark.Source: A MCP Servers · p. 20

• Tables 7–12 present 30 benchmark task examples, Tables 13–15 show three evaluator examples, and all tasks and evaluators are available on GitHub.Source: B Tasks and Evaluators Examples · p. 20

• All evaluated LLMs used temperature 1.0, with their versions listed in Table 16 and the experimental ReAct prompt shown in Figure 7.Source: C Setup · p. 20, Figure 7 · p. 28, Table 16 · p. 29
• Example benchmark tasks are provided for repository management, financial analysis, 3D designing, browser automation, and web searching.Source: Table 8 · p. 21, Table 9 · p. 22, Table 10 · p. 23
• Example evaluators are shown for location navigation, repository management, and financial analysis.Source: Table 13 · p. 25, Table 14 · p. 26, Table 15 · p. 27

D Naive Error

• o3 sometimes copies the prompt’s format requirements without doing anything, as shown in Figure 8.The paper characterizes this as a strange error for the model.Source: D Naive Error · p. 30

E Summarization Agent

• Figure 9 presents the summarization prompt used in the experiments.Source: E Summarization Agent · p. 30

F Exploration Agent

• The exploration-agent prompt used in the experiments is shown in Figure 10.Source: F Exploration Agent · p. 30, Figure 10 · p. 31