Source-linked AI summary
MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers
Chaithanya Bandi, Razvan-Gabriel Dumitru, Ben Hertzberg, Divyansh Agarwal, Geobio Boo, Tejas Polakam, Sami Hassaan, Jeff Da, HiJae Kim, Vipul Gupta, Manasi Sharma, Andrew Park, Martin Dimakis, Ernesto Gabriel Hernandez Montoya, Dan Rambado, Ivan Salazar, Rafael Cruz, MohammadHossein Rezaei, Chetan Rane, Ben Levin, Daniel Yue Zhang, Brad Kenstler, Bing Liu
TL;DR
Existing MCP evaluations lack a unified way to test realistic cross-server tool use across authentic servers while scoring answers reproducibly. MCP-Atlas addresses this with a large real-server benchmark and claim-level diagnostics, finding tiered frontier performance and predominantly cognitive failures. Its results are bounded by live-server drift, judge uncertainty, and provider-specific harness defaults.
Problem
Existing MCP evaluations fall short in realistic multi-step orchestration, authentic-server breadth, and reproducible claim-level scoring.
Method
MCP-Atlas evaluates natural-language tasks on real MCP servers using distractors, claim-based final-answer scoring, matched conditions, and an 11-category failure taxonomy.
Results
Across 20 frontier models, MCP-Atlas reveals a three-tier performance structure, with top models spanning 78.2–82.2% pass rate and nearly two thirds of failures cognitive.
Takeaways & Limitations
The benchmark shifts attention from valid API calls alone toward claim-aware stopping rules and trajectory-grounded answer verification.
Takeaways & Limitations
Results reflect live-server drift, judge uncertainty, provider-specific harness defaults, and bounded reproducibility of individual task trajectories.
Abstract
from arXiv · showhide
The Model Context Protocol (MCP) is emerging as a standard interface through which large language model (LLM) agents discover and invoke external tools. However, existing MCP evaluations fall short along three key axes: realistic multi-step workflows with cross-server orchestration, breadth across authentic MCP servers rather than mocks, and structured, reproducible claim-level scoring disentangled from agent verbosity or style. We introduce MCP-Atlas, a benchmark for measuring tool-use competency against production MCP servers. MCP-Atlas contains 1,000 natural-language tasks written and verified by human experts spanning 36 real MCP servers and 220 tools. Prompts do not specify servers, tools, or parameters, requiring agents to identify relevant tools among semantically plausible distractors and to compose multi-step, cross-server workflows. Each task is scored with a claim-level rubric, where final answers are scored against atomic factual claims grounded in tool outputs. This answer-centric scoring permits valid alternative tool-call trajectories to receive credit. We pair this with an 11-category diagnostic taxonomy that disentangles tool-call failures from cognitive failures in task understanding, synthesis, parsing, and stopping. Evaluating 20 frontier models from six providers under matched task-level conditions, we find pass rates up to 82.2% at a 0.75 claim coverage threshold and a clear three-tier performance structure. Automated diagnostics show that 63.3% of diagnosed failures are cognitive rather than tool-call related. Notably, several high-performing models fail after successful tool execution due to premature stopping or incorrect synthesis. We release the task schema, containerized harness, claim evaluator, and a 500-task public split, while reserving a 500-task private split to preserve leaderboard integrity. The code is at https://github.com/scaleapi/mcp-atlas.
1 Introduction
MCP-Atlas addresses gaps in MCP evaluation by combining realistic cross-server workflows, authentic server breadth, and claim-level scoring. Its evaluation of 20 frontier models reveals tiered performance and predominantly cognitive failures.
- Existing MCP benchmarks lack at least one of realistic orchestration, authentic-server breadth, and reproducible claim-level scoring.
- Claim-level scoring evaluates atomic factual claims and credits alternative tool-call trajectories that produce the same claims.
- 20 frontier models exhibit a three-tier structure, spanning 40.2% to 82.2% pass rates.
- MCP-Atlas evaluates 1,000 natural-language tasks across 36 production servers and 220 tools, with 98.6% requiring cross-server orchestration.
- 63.3% of diagnosed failures are cognitive, while 36.7% are tool-related; the taxonomy separates these into 11 categories.
2 Related Work
Prior evaluations expanded from static tests to interactive, tool-using benchmarks, but MCP-based work still trades off scale, realism, and objective scoring. MCP-Atlas combines these dimensions through manual verification, claims-based evaluation, real servers, and distractors.
- LLM evaluation progressed from fixed knowledge tests to interactive benchmarks for navigation, operating systems, planning, and tool use.
- MCP benchmarks expose a tension between rigorous small-scale evaluation and larger-scale evaluation that compromises objectivity or realism.
- MCP-Atlas uses 1,000 manually verified tasks across 36 real servers and 220 tools to achieve scale without automated task generation.
- Claims-based evaluation provides claim-level partial credit while avoiding trajectory matching and holistic style biases.
- Systematic distractors and exclusively real MCP servers test tool discovery under authentic server behavior rather than mocked conditions.
3 Benchmark Design
MCP-Atlas uses real, version-pinned MCP environments with distractor-heavy, cross-server tasks and claim-based scoring. Matched evaluation conditions, multiple judges, diagnostics, and layered quality assurance support reproducible assessment.
- MCP-Atlas uses 36 production servers exposing 220 tools, with live endpoints that preserve rate limits, pagination boundaries, schema mismatches, and transient errors.
- Each task exposes 6–37 tools, but only 2–8 are required; semantically similar distractors test tool discovery under noise.
- 98.6% of tasks require tools from at least two servers, while prompts omit server, tool, and parameter names.
- Each task averages 4.7 atomic claims scored as fulfilled, partial, or unfulfilled, with passing defined as coverage ≥0.75.
- Pass/fail scoring uses final-answer claims rather than reference trajectories, allowing alternative valid tool-call paths to receive credit.
- Models share prompts, exposed tools, budgets, rubrics, and protocols, while three independent judges and an 11-category taxonomy support evaluation and diagnosis.
4 Results and Analysis
Across matched evaluations of 20 models, MCP-Atlas reveals three performance tiers, predominantly cognitive failures, downstream bottlenecks among stronger models, and meaningful efficiency and judge-sensitivity considerations.
- 4.1 Overall Performance: 75.6%: open-source GLM-5.1 enters the same empirical band as several top proprietary models, while o3 Pro has no tool invocation in 40.1% of failed trajectories.The study describes GLM-5.1 as the strongest open model and identifies o3 Pro’s tool-use policy mismatch despite its reasoning performance.
- 4.2 Failure Mode Analysis: 63.3% of roughly 6,900 diagnosed failures are cognitive, versus 36.7% tool-related, indicating that remaining errors often occur after tool access.Tool-call failures remain especially relevant among lower-performing models.
- 4.2 Failure Mode Analysis: Early termination affects 42.8% of Gemini 3.1 Pro Preview’s failures but only 3.6% of GPT-5.5’s, whose faulty synthesis reaches 19.8%.Claude Opus 4.6 and 4.7 show the same downstream shift, with early termination at 36.3% and 3.8%, and faulty synthesis at 25.0% for Opus 4.7.
- 4.3 Efficiency Frontier: Six models lie on the pass-rate–time Pareto frontier, from Haiku 4.5 at 27 seconds and 40.2% to Muse Spark at 121 seconds and 82.2%.o3 Pro is strictly dominated at 194 seconds and 44.5%, while GLM-5.1’s 149-second average keeps it off the frontier despite strong accuracy.
- 4.4 Cross-Judge Agreement: Cross-judge score ranges span 2.1–4.6 percentage points, with rankings broadly preserved but small swaps concentrated in the crowded upper cluster.The public–private comparison reports gaps from −0.8 to +9.6 percentage points and treats provider clustering as an audit signal rather than contamination evidence.
5 Discussion
MCP-Atlas shows that real-server MCP evaluation is no longer chiefly limited by tool mechanics: the strongest models cluster near the top, yet substantial headroom remains in evidence completion and answer synthesis. Deployment-relevant evaluation must therefore consider diagnostics, latency, judge sensitivity, and ecosystem variability alongside pass rate.
- The top three models span 78.2–82.2% pass rate, while cross-judge ranges of 2.1–4.6 points make small ranking swaps uncertain.The strongest model still fails nearly one in five tasks, leaving headroom on realistic multi-server workflows.
- Diagnostic implications: Stronger agents often gather evidence correctly but stop before covering all claims or synthesize collected outputs incorrectly, shifting research attention beyond valid API calls.The paper specifically identifies claim-aware stopping rules and trajectory-grounded answer verification as targets.
- Deployment implications: MCP-Atlas exposes accuracy–latency domination, provider-clustered split gaps, live-server drift, judge uncertainty, and provider-specific harness defaults as deployment-relevant caveats.The authors recommend reporting diagnostic profiles that include accuracy, efficiency, judge sensitivity, split gap, drift, and failure mode.
6 Conclusion
MCP-Atlas presents a claims-scored benchmark of tool-use competency across production MCP servers and evaluates it across frontier models. It is intended both to track progress with remaining headroom and to diagnose where current agents fail.
- MCP-Atlas evaluates tool-use competency across 1,000 tasks, 36 production MCP servers, and 220 real tools using partial-credit claims-based scoring and three independent LLM judges.
- Figure 3 compares public and private performance across all 20 models, using large gaps as audit triggers rather than direct evidence of contamination.Kimi K2.5 is the only model whose private performance exceeds its public performance.
- MCP-Atlas is positioned as both a leaderboard with headroom for future agents and a diagnostic instrument for targeting failures in tool discovery, evidence gathering, synthesis, and stopping.
A Limitations
The benchmark’s conclusions are bounded by evaluator, ecosystem, language, harness, budget, claim-definition, diagnostic, and broader-impact considerations. These constraints affect reproducibility, comparability, scope, or interpretation of reported results.
- Judge-model ceiling: All evaluation uses LLM judges, so absolute pass rates should be interpreted within a 2–5 point cross-judge band and version changes may shift results.The authors recommend reporting and pinning the judge version.
- Snapshot of a moving ecosystem: Live servers and APIs change over time, so the May 2026 leaderboard is a moving snapshot and individual task trajectories cannot be fully reproduced from upstream behavior alone.Container pinning and re-scoring runs mitigate, but do not eliminate, this boundary.
- English-only prompts: All 1,000 tasks are English-only and target English-language server responses, excluding non-English tool use, script handling, and localization failures.
- Provider-specific harness defaults: Provider-recommended native tool-calling strategies make the harness follow published best practices but not fully identical across providers.
- Tool-call budget: A fixed 100-call budget causes some failures, particularly on long-horizon retry-heavy tasks, although internal tests have not shown changed relative rankings when relaxing it.
- Claim-extraction error band: Residual ambiguity in manually reviewed claim lists adds noise to achievable pass rates, including for a hypothetical perfect agent.
- Reference Trajectory in Diagnostics: Reference-trajectory diagnostics may reflect one author-provided evidence path when multiple substantially different paths exist, despite claim-based scoring separating pass/fail from that trajectory.
- Broader impacts and risks: The benchmark supports safer and more accountable agent development while also potentially accelerating agents that interact with external systems, creating broader-impact risks.
B Server and Tool Inventory
MCP-Atlas inventories 36 production MCP servers exposing 220 tools, with tasks distributed across application buckets and server usage varying by task. Its evaluation combines real-server execution with claim-level scoring and confidence-interval reporting.
- Server and Tool Inventory: 36 production MCP servers expose 220 distinct tools, with each server deployed in a pinned container and exercised against live endpoints.Per-server catalogs, container image digests, and version pins accompany the dataset.
- Server and Tool Inventory: 32% of tasks are BASIC, 12% ANALYTICS, 22% PRODUCTIVITY, 12% FINANCIAL, and 22% CODING.
- Server and Tool Inventory: Server usage varies across the benchmark, with general-purpose search and filesystem servers more heavily represented.
- Server and Tool Inventory: The server selection criteria emphasize stable implementations, functional diversity, related tools for distractor selection, and realistic multi-step workflows.
- Server and Tool Inventory: Containers use sandboxed filesystems, allow-listed network egress, task-level restarts, and pinned image digests to limit state leakage and improve reproducibility.
- Evaluation Scoring: Each ground-truth claim receives an independent categorical judge score: fulfilled maps to 1.0, partially_fulfilled to 0.5, and not_fulfilled to 0.0.
- Evaluation Scoring: Task-level coverage averages claim scores, and the main analyses classify tasks as passing at coverage ≥0.75.
- Evaluation Scoring: 95% confidence intervals for pass@0.75 use 10,000 bootstrap resamples of N = 1000 tasks.
D Task Authoring Templates and Example Task
Task authoring combines expert verification, natural-language prompts, curated relevant tools and distractors, and grounded deliverables. The example illustrates how multi-server evidence becomes atomic claims in a final answer, while failure analysis separates authoring checks from model diagnostics.
- D Task Authoring Templates and Example Task: Each task required an average of 4.6 hours of expert labor for authoring, verification, and documentation.
- D Task Authoring Templates and Example Task: The full internal authoring manual remains unreleased because it contains proprietary procedures, access-control details, and private benchmark templates.
- D.1 Task Authoring Requirements: Prompts were required to depend on external data, remain stable during evaluation, use natural and unambiguous language, and avoid revealing solution paths.
- D.1 Task Authoring Requirements: Authors were encouraged to create reasoning-intensive multi-step requests involving indirect references or filtering across multiple sources.
- D.2 Tool Environment Construction: Tool environments provide minimally sufficient relevant tools, plausible distractors, and cross-family evidence for multi-source tasks without unnecessary ambiguity.
- D.3 Model Evaluation and Failure Taxonomy: The coarse authoring checklist covers tool selection, tool input, and interpretation, distinct from the final 11-mode automated failure taxonomy.
- D.3 Model Evaluation and Failure Taxonomy: The taxonomy distinguishes tool selection, tool input, and interpretation errors as separate failure categories.
- D.4 Required Deliverables: Each task includes a reference trajectory with rationales and a grounded final answer supported strictly by tool outputs.
TASK CONTEXT:
The diagnostic prompt compares expected and actual trajectories, missed claims, and final responses to assign a specific failure mode. It distinguishes tool interaction problems from reasoning and synthesis problems using explicit boundary rules.
- TASK CONTEXT:: The diagnostic context supplies task identifiers, original prompts, coverage scores, expected trajectories and claims, agent traces, final responses, and scorer reasoning.
- TASK CONTEXT:: The taxonomy separates tool-call failures such as malformed calls, wrong tools, and no tool use from cognitive failures such as misunderstanding and faulty synthesis.
- TASK CONTEXT:: Diagnosis proceeds by anchoring expected behavior, examining actual behavior, considering missed claims, and mapping the gap to a failure mode.
- TASK CONTEXT:: A response misparsing error reads the wrong field or row, whereas faulty synthesis combines correctly read outputs incorrectly.
- TASK CONTEXT:: A logical error is a flawed multi-step reasoning chain, while faulty synthesis is incorrect combination without a clear logic error.
- TASK CONTEXT:: Hallucinated facts are absent from tool outputs, whereas no_tool_use bypasses required tools entirely for a fact.
- TASK CONTEXT:: Malformed_call identifies the first incorrect parameter use, while err_recovery identifies repeated identical errors without adaptation.
- TASK CONTEXT:: The judge selects one primary mode, records contributing failures, marks root causes, calibrates confidence, and summarizes the missed claim with trajectory evidence.
F Cross-Judge Agreement (Extended)
Across 20 models, coverage-threshold rankings are largely stable, but stricter passing criteria widen the gap between strong and weak systems. This supports reporting a single headline threshold while retaining multi-threshold detail.
- F Cross-Judge Agreement (Extended): Table 7 compares each model’s pass rate at coverage ≥0.75 across three evaluator judges, including cross-judge ranges and ranks.
- F Cross-Judge Agreement (Extended): Table 8 reports mean coverage and pass rates at 0.50, 0.75, and 0.9 for all 20 models.
- F Cross-Judge Agreement (Extended): ρ = 0.965 is the Spearman rank correlation between pass rates at 0.50 and 0.75 across the 20 models.
- F Cross-Judge Agreement (Extended): 89.6 →82.2 is the pass-rate change for Muse Spark from coverage 0.50 to 0.75, a 7.4 pp drop.
- F Cross-Judge Agreement (Extended): 55.8 →40.2 is the pass-rate change for Claude Haiku 4.5 from coverage 0.50 to 0.75, a 15.6 pp drop.
H Access, Licensing, and Reproducibility
MCP-Atlas formalizes tasks as controlled interactions over exposed MCP tools, with claims-based scoring and reference trajectories supporting reproducibility and diagnostics rather than pass/fail matching.
- Task representation: Each public task row includes an identifier, enabled tools, natural-language prompt, grounded claims, and a serialized reference trajectory.The trajectory records tool calls, arguments, dependencies, and returned evidence sufficient to solve the task.
- Task representation: Reference trajectories support reproducibility, smoke tests, solvability audits, and post-evaluation diagnostics, but are not matched for pass/fail scoring.
- Release and reproducibility: The release provides the public split, schema, claims, evaluation harness, claims evaluator, pinned dependencies, reference wrapper, regeneration scripts, and model configurations.The private split remains unreleased for contamination monitoring and leaderboard integrity; licensing differs across dataset, code, and upstream server implementations.
- Formal task definition: A task specifies a goal, natural-language constraints, an allow-listed tool set, a diagnostic reference trajectory, and independently verifiable answer claims.The allow-listed set may include required target tools and semantic distractors.
- Interaction and scoring: Agents issue typed tool calls in a single-turn interaction, receive values or errors, and must finally emit a textual answer.The interaction is modeled as a finite-horizon POMDP with rewards assigned afterward by claims-based scoring.
- Interaction and scoring: Pass/fail depends solely on the rubricized correctness of the final answer against the claims list, independent of the reference trajectory.