Source-linked AI summary
MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers
Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, Junnan Li
TL;DR
Existing benchmarks do not adequately evaluate LLMs in realistic MCP environments with long-horizon interactions and unfamiliar tools. MCP-Universe addresses this gap with a multi-domain benchmark using real-world MCP servers and execution-based evaluators; even leading models show substantial limitations, while long-context handling remains a major challenge.
Problem
Existing evaluations are insufficiently realistic for assessing LLM capabilities with MCP, omitting challenges such as long-horizon reasoning and large, unfamiliar tool spaces.
Method
MCP-Universe benchmarks 231 tasks across six domains and 11 real-world MCP servers using format, static, and dynamic execution-based evaluators.
Results
Leading models remain limited in MCP environments: GPT-5 achieves the highest overall success rate at 43.72%, followed by Grok-4 at 33.33% and Claude-4.0-Sonnet at 29.44%.
Takeaways & Limitations
The benchmark exposes long-context handling, unknown-tool usage, and cross-domain performance disparities as fundamental challenges for current LLM agents.
Takeaways & Limitations
Long context handling is particularly challenging in Location Navigation, Browser Automation, and Financial Analysis, where lengthy observations or historical actions can exceed model context windows.
Abstract
from arXiv · showhide
The Model Context Protocol has emerged as a transformative standard for connecting large language models to external data sources and tools, rapidly gaining adoption across major AI providers and development platforms. However, existing benchmarks are overly simplistic and fail to capture real application challenges such as long-horizon reasoning and large, unfamiliar tool spaces. To address this critical gap, we introduce MCP-Universe, the first comprehensive benchmark specifically designed to evaluate LLMs in realistic and hard tasks through interaction with real-world MCP servers. Our benchmark encompasses 6 core domains spanning 11 different MCP servers: Location Navigation, Repository Management, Financial Analysis, 3D Design, Browser Automation, and Web Searching. To ensure rigorous evaluation, we implement execution-based evaluators, including format evaluators for agent format compliance, static evaluators for time-invariant content matching, and dynamic evaluators that automatically retrieve real-time ground truth for temporally sensitive tasks. Through extensive evaluation of leading LLMs, we find that even SOTA models such as GPT-5 (43.72%), Grok-4 (33.33%) and Claude-4.0-Sonnet (29.44%) exhibit significant performance limitations. In addition, our benchmark poses a significant long-context challenge for LLM agents, as the number of input tokens increases rapidly with the number of interaction steps. Moreover, it introduces an unknown-tools challenge, as LLM agents often lack familiarity with the precise usage of the MCP servers. Notably, enterprise-level agents like Cursor cannot achieve better performance than standard ReAct frameworks. Beyond evaluation, we open-source our extensible evaluation framework with UI support, enabling researchers and practitioners to seamlessly integrate new agents and MCP servers while fostering innovation in the rapidly evolving MCP ecosystem.
1 Introduction
MCP-Universe addresses limitations in existing evaluations by benchmarking LLMs on realistic tasks using real-world MCP servers and execution-based assessment. It spans six domains and reveals that current models face substantial performance, long-context, and unknown-tools challenges.
- MCP-Universe evaluates LLMs in realistic, challenging use cases through interaction with real-world MCP servers.
- The benchmark covers six domains, 11 MCP servers, and 231 tasks involving operationally complex applications.Its domains include Location Navigation, Repository Management, Financial Analysis, 3D Design, Browser Automation, and Web Searching.
- Execution-based evaluators measure format compliance, time-invariant content, and temporally sensitive tasks using real-time ground truth.
- 43.72% success rate is achieved by GPT-5, while Grok-4 reaches 33.33% and Claude-4.0-Sonnet 29.44%, showing substantial performance limitations.
- LLM agents encounter long-context and unknown-tools challenges, alongside marked variation in success across application domains.Interaction steps can rapidly increase token counts, while agents may lack familiarity with MCP tool usage patterns, parameters, and expected behaviors.
2 Related Work
Prior agent benchmarks cover web navigation, GUI interaction, and related MCP scenarios, but MCP-related evaluations differ in realism, temporal coverage, and application relevance. MCP-Universe is positioned against these varied evaluation settings.
- MCP research builds on agent capabilities for autonomous planning, reasoning, and tool use, with patterns including ReAct, Reflection, and Plan-and-Solve.
- Existing agent benchmarks evaluate web navigation and GUI interaction through environments such as WebArena, OSWorld, and WindowsAgentArena.
- MCPWorld uses real-world GUIs but lacks time-varying tasks, while MCP-RADAR adapts existing datasets whose tasks are less related to real-world applications.
3 MCP-Universe
MCP-Universe is an extensible benchmark that evaluates LLM agents on practical tasks using real-world MCP servers and automated execution-based assessment. It combines dynamically configured agent-server interactions with diverse domains, tools, and task designs that require tool selection, sequencing, and adaptation.
- Evaluation Framework: MCP-Universe combines an extensible evaluation framework, real-world task instructions, and execution-based evaluators to measure task completion.The framework dynamically configures agents, servers, and evaluators for each task specification.
- Task Formulation: Each task specifies a goal, initial context, and the tools available through selected MCP servers.The benchmark models tasks as tuples (G, C, Tavailable), where accessible tools are drawn from the selected servers.
- Task Formulation: Agents must identify, sequence, and invoke appropriate tools while reasoning over partial information and adapting to diverse interfaces or failures.Success is assessed from interaction traces using predefined automated checks, producing a binary task-completion outcome.
- Task Construction: MCP-Universe contains manually designed challenging tasks covering multiple usage types within each application domain.Tasks were created to reflect real use cases and filtered when they were easily solvable without MCP or consistently solvable within five retries.
- Tasks and Evaluators: Its evaluators comprise format, static, and dynamic checks for compliance, time-invariant correctness, and temporally sensitive ground truth.The execution-based design avoids relying solely on LLM judges for tasks involving real-time data.
4 Experiment
Experiments evaluate proprietary and open-source LLMs and agent configurations across MCP-Universe’s domains, revealing low overall success, evaluator-specific weaknesses, long-context and unknown-tool challenges, and framework-dependent performance.
- Frontier Models Performance: 43.72% overall success rate makes GPT-5 the strongest evaluated model, ahead of Grok-4 at 33.33% and Claude-4.0-Sonnet at 29.44%.GPT-5 is particularly strong in Financial Analysis and 3D Designing, while Location Navigation remains difficult for all models.
- Frontier Models Performance: Execution-based evaluation separates format compliance, static content matching, and dynamic real-time-grounded performance.Format evaluators measure agent format compliance, while static and dynamic evaluators assess content under time-invariant and temporally sensitive conditions.
- Frontier Models Performance: Content evaluation is harder than format compliance: many models exceed 80% on format evaluators, while most achieve only 40–60% on content evaluators.Claude-4.0-Sonnet leads format evaluation at 98.29%, while GPT-5 and Claude-4.0-Sonnet lead static evaluation at 61.92%; GPT-5 leads dynamic evaluation at 65.96%.
- Long Context Challenges: Context length increases rapidly with interaction steps, creating a major challenge in Location Navigation, Browser Automation, and Financial Analysis.Google Maps, Playwright, and Yahoo Finance MCP servers can return extensive location, HTML, and historical stock information.
- Long Context Challenges: Summarization is insufficient as a general solution: it has no effect or harms performance in some tested domains.The summarization agent compresses raw MCP outputs at each step, but Claude-4.0-Sonnet sees no benefit in Location Navigation and negative effects in Browser Automation and Financial Analysis.
- Unknown Tools Challenges: Unknown tool interfaces cause execution errors, while exploration improves some model-domain combinations but not others.GPT-4.1 gains 7.69 percentage points in Browser Automation, and Claude-4.0-Sonnet gains 7.50 percentage points in Financial Analysis; other combinations decline or remain unchanged.
- Tool Complexity: Connecting 7 MCP servers with 94 tools adds noise and reduces performance, including Claude-4.0-Sonnet’s Location Navigation success rate falling from 22.22% to 11.11%.The expanded configuration connects unrelated MCP servers rather than only task-relevant servers.
- Enterprise-Level Agent Framework Comparison: ReAct outperforms Cursor Agent with the Claude-4.0-Sonnet backbone overall, while OpenAI Agent SDK outperforms ReAct with the o3 backbone.ReAct reaches 29.44% versus Cursor Agent’s 26.41%; OpenAI Agent SDK reaches 31.60% versus ReAct’s 26.41%.
5 Conclusion
MCP-Universe evaluates LLMs in real-world MCP environments using authentic data and execution-based evaluators. Experiments expose gaps in long-context handling, tool familiarity, and cross-domain performance, motivating advances in models and agent integration.
- Conclusion: MCP-Universe is a comprehensive benchmark for rigorously evaluating LLMs in real-world MCP environments.Its extensible framework and user-friendly interface are intended as a testbed for researchers and practitioners.
- Conclusion: Experiments expose critical gaps in long-context handling, tool familiarity, and cross-domain performance among current LLMs and enterprise-level agents.The benchmark grounds tasks in authentic data and uses execution-based evaluators.
A MCP Servers
The benchmark documents its MCP servers and provides links to support reuse. Most servers are official, while others are based on official APIs.
- MCP Servers: Most MCP servers in the benchmark are official, and the remaining servers are based on official APIs to ensure server quality.Table 6 includes server names and links to make them easier to use.
B Tasks and Evaluators Examples
The paper provides examples of benchmark tasks and evaluators, with all tasks and evaluators available in the project’s GitHub repository.
- Tasks and Evaluators Examples: Tables 7–12 contain 30 task examples, while Tables 13–15 contain three evaluator examples.The paper states that all tasks and evaluators can be found on GitHub.
C Setup
The setup defines benchmark task examples, evaluator examples, model configurations, and ReAct prompting for location-navigation experiments. The prompts require tool selection, stepwise reasoning, JSON-formatted responses, and completion within 20 steps.
- The experiments use specified LLM versions with temperature fixed at 1.0.
- The benchmark setup includes example tasks for repository management, financial analysis, 3D design, browser automation, and web searching.
- Evaluator examples cover location navigation, repository management, and financial analysis.
- The setup includes a branch-existence checking function that lists repository branches and verifies each requested branch name.
- The location-navigation prompt instructs the agent to select an explicitly defined tool when needed and otherwise answer directly.
- The ReAct agent analyzes the query and previous results, chooses between tool use and a final answer, and must respond in JSON within 20 steps.
D Naive Error
The paper presents a naive error example in which o3 copies prompt-format requirements instead of performing the requested task.
- o3 sometimes directly copies the prompt’s format requirements without doing anything.The paper characterizes this behavior as a strange error for the model.
E Summarization Agent
The paper presents the summarization prompt used in its experiments.
- The experiments include a dedicated summarization prompt.
F Exploration Agent
The exploration agent uses a separate exploration phase to learn tool capabilities and collect task information before exploitation. It tracks phase-specific iteration budgets and accumulated tool knowledge, then permits final answers during exploitation.
- F Exploration Agent: The exploration agent is a two-phase Explore and Exploit agent that first learns tool capabilities and then uses that knowledge to solve tasks.
- F Exploration Agent: The prompt exposes the current phase and remaining exploration iterations to the agent.
- F Exploration Agent: During exploration, the agent focuses on learning tool capabilities, effects, and task information rather than task completion.
- F Exploration Agent: The exploration phase forbids final answers and requires the agent to use tools to explore and learn.
- F Exploration Agent: Exploitation is constrained by a remaining-iteration budget and requires a final answer when no iterations remain.
- F Exploration Agent: The agent records previous reasoning steps and observations alongside accumulated tool knowledge.
- F Exploration Agent: The prompt instructs the agent to analyze prior observations, choose a tool or final response, and follow a JSON format.