MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools

1 Introduction

• MCP is presented as a universal communication layer supporting server-level context or state, richer feedback, and dynamic tool discovery.These properties are contrasted with traditional function calling [36].Source: §1 · pp. 1–2
• Existing function-calling benchmarks [36] omit standardized MCP interactions and may not reliably differentiate agent capabilities.The introduction also describes an ecosystem-level M×N integration problem across agents and disparate tools.Source: §1 · pp. 1–2
• MCP-AgentBench uses a unified testbed of 33 MCP servers and 188 tools for realistic, standardized evaluation.Servers were selected for executability, statelessness, and text-based interaction.Source: §1 · p. 2
• MCP-Eval uses an LLM-as-a-judge methodology to assess tangible task success rather than adherence to a fixed execution path.The method recognizes that complex problems can have multiple valid solution trajectories.Source: §1 · p. 2
• 600 queries span 6 categories and interaction complexities, from single-server operations to multi-server sequential workflows.The construction process also includes structured query generation and LLM-assisted, human-verified reference answers.Source: §1 · p. 2
• Evaluations of leading agents provide foundational insights into current capabilities, limitations, and challenges in mastering MCP.Source: §1 · p. 2

2 MCP-AgentBench

• The benchmark section presents the dataset construction pipeline, dataset statistics, and MCP-Eval as its framework for assessing agent performance.The workflow figure depicts testbed establishment, query generation, reference-answer production, and pass/fail evaluation of final answers.Source: §2 · p. 2, Figure 2 · p. 3

2.1 Data Construction Methodology

• The construction pipeline has three stages—server testbed, query generation, and reference-answer annotation—with LLM assistance and human verification at each stage.Claude 3.7 Sonnet assists throughout the process.Source: §2.1 · p. 3

2.1.1 Establishing the MCP Server Testbed: Rigorous Curation, Deployment, and Integration

• The testbed retains 33 of 369 surveyed servers, prioritizing stable, stateless, text-based systems, and unifies their access to 188 tools through mcprouter.Stateful servers were excluded to support reproducible assessment; non-text modalities were excluded because quantitative evaluation was complex.Source: §2.1.1 · pp. 3–4

2.1.2 Query Generation

• Queries span six categories formed by crossing single- versus multi-server scope with single, parallel, or sequential tool calls.LLM-generated contextual components are synthesized into self-contained, goal-oriented user queries and then human-verified.Source: §2.1.2 · pp. 4–5

2.1.3 Reference Answer Annotation

• Reference answers use a hybrid expert-in-the-loop process that screens all queries automatically and escalates low-pass-rate cases for expert review.Experts may revise flawed queries, improve answers, or author references; trajectories are not treated as ground truth because tasks may have multiple valid paths.Source: §2.1.3 · p. 5

2.2 Data Statistics

• 600 queries are evenly divided across six interaction categories, with server selection randomized in multi-server categories to keep the tool count near a 128-tool limit.Each category contains 100 queries.Source: §2.2 · p. 5

2.3 MCP-Eval

• MCP-Eval uses an LLM judge to score final task outcomes, prioritizing functional success over execution paths and superficial formatting.The judge applies principles including user intent, sufficient answers, and evidence of external data when tool use is required.Source: §2.3 · pp. 5–6

3.1 Experimental Setup

• Ten representative proprietary and open-source LLMs were evaluated through official APIs, using ReAct and native tool calling where supported.Gemini’s tool-calling mode was excluded because deployed-server tool descriptions were incompatible with its parser; each query was limited to 30 actions.Source: §3.1 · p. 6

3.2 Main Results

• Qwen3-235B-A22B with ReAct achieved the benchmark’s highest overall score; Claude 4 Sonnet led proprietary models, while GPT-4o underperformed across tested scenarios.Kimi K2 performed particularly well in TC mode, and o3-mini was consistent across both modes.Source: Table 1 · p. 6, §3.2 · p. 7
• No interaction mode was universally superior: Qwen3 led with ReAct but collapsed in TC, while Claude 4 Sonnet improved with TC.Qwen3 often failed to issue a required tool call in TC, causing premature termination; subsequent analysis used native TC results for models supporting it.Source: §3.2 · p. 7, §3.3 · p. 7

3.3 Analysis

• Pass rates generally declined from single- to multi-server tasks and from single to sequential calls, although Claude 4 Sonnet improved on multi-server tasks.The authors attribute this exception to greater complexity compelling tool use and mitigating Claude 4 Sonnet’s tendency to rely incorrectly on parametric knowledge.Source: §3.3 · p. 7, Figure 4 · p. 7
• Kimi K2 and Claude 4 Sonnet paired the highest pass rates, 61.0% and 58.0%, with 101.7k and 140.3k tokens per query, respectively.o3-mini achieved a 50.0% pass rate at 36.5k tokens; the authors attribute much of the higher consumption to internal reasoning in “thinking” mode.Source: §3.3 · p. 7, Figure 5 · p. 7
• MCP-Eval agreed with the human majority on 91.67% of judgments (Cohen’s Kappa 0.734); expert pass/fail inter-rater Fleiss’ Kappa was 0.671.Three experts annotated 60 Claude 3.7 Sonnet outputs, sampling 10 items from each of six categories.Source: §3.3 · pp. 7–8
• Recurring errors included hallucination, query misinterpretation, refusal to use required tools, and omission of key information from multi-step responses.Hallucination involved unsupported or contradictory claims; tool refusal involved answering from parametric knowledge when external data was required.Source: §3.3 · p. 8

LLM Agents and Standardized Interaction

• MCP provides a universal communication layer between agents and tools, addressing the “M×N integration problem” as tool interactions scale.The adoption of standardized protocols creates a need for benchmarks designed for structured environments.Source: §4 · p. 8

Benchmarking Protocol-Aware Tool Use

• General tool-use benchmarks such as ToolBench and API-Bank were not designed to assess effectiveness within standardized protocols.MCP-Bench evaluates servers rather than agent proficiency, while MCPEval and MCP-RADAR have limited server or domain scope.Source: §4 · p. 8
• The related-work review identifies a gap for large-scale evaluation of agents across diverse, operational MCP infrastructure.MCP-AgentBench is introduced to address this gap.Source: §4 · p. 8

5 Conclusion

• The MCP Server Testbed provides an operational environment with 33 servers and 188 tools.It is described as realistic and standardized.Source: §5 · p. 8
• MCP-AgentBench comprises 600 diverse queries for assessing agent performance on the testbed.Source: §5 · p. 8
• MCP-Eval is an outcome-oriented LLM-as-a-judge methodology that prioritizes tangible task success.The conclusion frames the combined work as a standardized framework with empirical insights for advancing capable, interoperable AI systems.Source: §5 · p. 8

A Data Statistics

• Query lengths are right-skewed and peak at 800–1,000 characters, while longer, more variable reference answers peak at 2,000–4,000 characters.The distributions show an asymmetry between moderately detailed inputs and more extensive outputs.Source: A Data Statistics · p. 11, Figure 6 · p. 11

B Reference Answer Annotation Details

• Reference-answer annotation used five LLM-generated execution trajectories per query; queries below a 20% aggregate pass rate were sent for expert review.Source: B Reference Answer Annotation Details · p. 11

C.1 ReAct Format

• ReAct history records the user query, successive thought-action-observation cycles, and the final answer; exceeding the step limit prompts an immediate final answer.The ordered history serves as contextual input for subsequent LLM prompts.Source: §C.1 · p. 11

C.2 Tool Calling Mode

• Tool Calling alternates between model tool calls and returned observations until the model produces a final answer without further calls.Some models issue one call per step, while GPT-4o can issue multiple calls concurrently.Source: §C.2 · p. 11

C.3 Configuration

• Enhanced-reasoning models used an 8192-token thinking budget, generations were capped at 8192 tokens, and transient API failures triggered up to three retries.Source: §C.3 · p. 12

• The consistency metrics were applied to 60 items classified into two categories by MCP-Eval and three human experts.Source: D Calculation of Metrics for… · p. 12

• Percentage agreement is the proportion of items on which MCP-Eval’s decision matches the human expert majority vote.Source: §D.1 · p. 12

• Cohen’s Kappa measures agreement between MCP-Eval and the human majority vote while correcting for chance agreement.Source: §D.2 · p. 12
• Chance agreement is calculated by summing, across categories, the product of each category’s assignment proportion for MCP-Eval and the human majority.The expected agreement probability uses the proportion of items each rater assigns to each category.Source: §D.2 · p. 12

D.3 Cohen’s Kappa (MCP-Eval vs. Human Majority)

• Kappa is calculated as (P̄ − P̄e) / (1 − P̄e) to measure agreement beyond chance.The measure is applied to N items, two categories, and three human experts.Source: §D.3 · p. 12
• P̄ is the mean observed proportion of rater agreement per item, with each item’s agreement based on category assignments by the raters.For item i, n_ij counts raters assigning it to category j.Source: §D.3 · p. 12
• P̄e is expected chance agreement, calculated from the proportions of all assignments falling into each category.The category proportions are based on the counts n_ij across items and raters.Source: §D.3 · p. 12

D.4 Overall Three-Way Percentage Agreement (Among Experts)

• Three-way agreement is the proportion of N items for which all three human experts gave an identical rating.The measure is labeled Nall_agree.Source: §D.4 · p. 12

Query Synthesis System Prompt

• Generated queries must be single-turn, authentic user requests that are fully solvable with selected tools and contain enough information for deterministic completion.They should naturally express the user’s objective and category pattern without mentioning tools or simulation mechanics.Source: Query Synthesis System Prompt · pp. 13–14

Query Synthesis User Prompt

• The query-generation prompt supplies system time, category, user profile, scenario, objective, MCP servers, and selected tools as inputs.These values are provided in labeled fields.Source: Query Synthesis User Prompt · pp. 14–15

CRITICAL REQUIREMENTS

• Queries must follow the specified single- or multi-server call pattern, be solvable only with selected tools, and form one complete, profile-authentic request.The prompt distinguishes single, parallel, and sequential interaction patterns.Source: CRITICAL REQUIREMENTS · p. 15

Contextual Component Generation System Prompt

• Scenario design scales from simple single-call tasks to complex cross-server sequential workflows, with deterministic, traceable outcomes and temporal context when relevant.The prompt lists six categories spanning single- and multi-server calls, parallel calls, and dependency-driven sequences.Source: Contextual Component Generation System… · pp. 15–16

1. Server and Tool Selection

• Tool selection and scenario construction must match category complexity, use deterministic and objectively verifiable capabilities, and ensure all parameters are available without follow-up questions.Design also specifies a consistent user persona, concrete constraints, and category-appropriate success criteria.Source: §1 · pp. 17–18

Contextual Component Generation User Prompt

• The contextual-component prompt asks for selected tools, user profile, scenario, and objective that jointly match the requested call pattern and yield deterministic, verifiable outcomes.Single-call objectives are straightforward, parallel objectives independent, and sequential objectives dependency-driven.Source: Contextual Component Generation User… · p. 18

ReAcT Assistant Prompt

• The ReAcT assistant should choose tools strategically, parallelize independent calls, sequence dependent calls, and synthesize results into a clear response.Tool parameters must be supported by the query, history, prior results, or reasonable defaults; the assistant should answer directly when tools add no value.Source: ReAcT Assistant Prompt · pp. 19–20
• The ReAcT assistant prompt asks responses to integrate relevant history, answer completely and logically, highlight key insights, and use the user’s language.The tool-calling user prompt supplies the query in a user_query field.Source: ReAcT Assistant Prompt · p. 20, Tool Calling Assistant User Prompt · p. 20

Tool Calling Assistant System Prompt

• The system prompt directs the assistant to choose relevant, efficient MCP tools, parallelize independent calls, handle failures, and synthesize results clearly.It also permits direct answers when tools would add no relevant information and instructs the assistant not to invent unsupported parameters.Source: Tool Calling Assistant System Prompt · p. 21

Evaluation Prompt

• Evaluation passes when specific external data addresses the user’s core need; generic knowledge alone fails, while extra detail and source attribution are not required.The rubric judges sufficiency against the query’s primary purpose, not completeness or conformity to the reference answer.Source: Evaluation Prompt · pp. 22–24
• The evaluation process compares tool-use evidence and whether the core need is met, accepting reasonable differences in answer details or format.The reference answer indicates the needed tool type, possible execution level, and query scope; it is not the only acceptable response.Source: Evaluation Prompt · pp. 23–25