PaperlayerView exact revision on arXiv

Source-linked AI summary

LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?

Guozhao Mo, Wenliang Zhong, Jiawei Chen, Qianhao Yuan, Xuanang Chen, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, Le Sun

arXiv:2508.01780v2cs.AIcs.CL

TL;DR

Existing MCP benchmarks provide limited evaluation of large-scale retrieval and multi-tool composition, particularly in dynamic settings. LiveMCPBench tests these capabilities on daily tasks with a reproducible toolset and outcome-based evaluation; results show a substantial model performance gap and point to retrieval as a primary bottleneck.

  • Problem

    Existing MCP benchmarks are narrow or static and may inject tools directly into context, leaving large-scale retrieval and multi-tool composition insufficiently evaluated.

    Source: §1 · pp. 1–2

  • Method

    LiveMCPBench evaluates 95 multi-step daily tasks with a ready-to-use collection of 70 MCP servers and 527 tools, using an LLM judge to verify outcomes in dynamic settings.

    Source: §1 · p. 2, §2.3 · p. 4

  • Results

    78.95% task success for Claude-Sonnet-4, while most widely used frontier models remain in the 30—50% range; nearly half of failures stem from tool retrieval errors.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study identifies tool retrieval and effective composition as unresolved challenges for agents operating in large-scale MCP settings.

    Source: §6 · p. 8

    LiveMCPEval relies heavily on LLM-based evaluation, so potential model biases may still influence results despite human validation.

    Source: A Limitations · p. 10

Abstract

from arXiv · show

Model Context Protocol (MCP) has become a key infrastructure for connecting LLMs with external tools, scaling to 10,000+ MCP servers with diverse tools. Unfortunately, there is still a large gap between real-world MCP usage and current evaluation: they typically assume single-server settings and directly inject tools into the model's context, bypassing the challenges of large-scale retrieval and multi-tool composition. To bridge this gap, we propose LiveMCPBench, which evaluates 95 real-world daily tasks explicitly constructed to stress diverse tools and scaled multi-server routing. The benchmark includes a ready-to-deploy tool suite of 70 servers with 527 tools, ensuring reproducibility without scattered API configuration. We further introduce an LLM-as-a-Judge evaluation framework that directly verifies task outcomes, handling dynamic data sources and multiple valid solution paths. We benchmark 12 state-of-the-art LLMs and observe a substantial performance gap: while Claude-Sonnet-4 reaches 78.95% task success, most models achieve only 30-50%. Our analysis reveals that the active tool composition strongly correlates with task success, whereas retrieval errors account for nearly half of all failures, highlighting retrieval as the dominant bottleneck. Together, these results provide the first large-scale, reproducible diagnosis of MCP agent capabilities and point towards future research on improving retrieval robustness and encouraging effective tool composition. Our code and data are publicly available at https://icip-cas.github.io/LiveMCPBench.

1 Introduction

LiveMCPBench addresses evaluation gaps in a rapidly scaling MCP ecosystem by testing retrieval and tool composition on daily tasks. Its evaluation of 12 models finds a substantial performance gap and identifies retrieval errors as a primary bottleneck.

1 Introduction

  • MCP has become core infrastructure for connecting LLMs to external tools, with an ecosystem exceeding 10,000 servers.The introduction presents effective MCP use as an important measure of agent capability.Source: §1 · p. 1
  • Existing MCP benchmarks are narrow or static, and some inject tools directly into context, bypassing large-scale retrieval and multi-tool composition.The section frames systematic retrieval and composition evaluation, alongside scalable and reproducible evaluation in dynamic settings, as central research questions.Source: §1 · pp. 1–2
  • LiveMCPBench evaluates 95 multi-step daily tasks; its ready-to-use toolset contains 70 servers and 527 tools, and LiveMCPEval uses an LLM judge for dynamic tasks and multiple valid trajectories.The framework is designed to support open-ended expansion without costly manual annotation.Source: §1 · p. 2
  • 78.95% task success for Claude-Sonnet-4, while most widely used frontier models remain in the 30—50% range; nearly half of failures stem from tool retrieval errors.Active tool composition also strongly correlates with task success.Source: §1 · p. 2

2 LiveMCPBench

LiveMCPBench evaluates agents’ ability to retrieve appropriate tools for general-purpose daily tasks in large-scale MCP settings. Its design combines dynamic, multi-step tasks, a reproducible toolset, outcome-based evaluation, and an adaptive agent baseline.

2 LiveMCPBench

  • The benchmark targets tool retrieval for general-purpose daily tasks and is designed around representative tasks, a reproducible toolset, and automated evaluation of dynamic tasks.These are the three design challenges the section identifies.Source: §2 · p. 2

2.1 Task Construction

  • 95 tasks across six daily domains were developed through student-proposed scenarios and validation, targeting dynamic, long-horizon needs involving coordinated tools.The domains include Office, Lifestyle, Leisure, Finance, Travel, and Shopping.Source: §2.1 · p. 3

2.2 LiveMCPTool Collection

  • The curated, Docker-packaged toolset contains 70 servers and 527 tools, selected through filtering, categorization, and manual vetting for standalone functionality and relevance.The collection process begins with 5,588 marketplace server configurations and filters out servers requiring keys or third-party integrations.Source: §2.2 · pp. 3–4

2.3 LiveMCPEval

  • LiveMCPEval uses an LLM judge to verify task completion from task key points, tool-use patterns, and contextual feedback rather than matching a fixed trajectory.It returns binary success or failure and is designed to expand with new servers and tasks without costly manual annotation.Source: §2.3 · p. 4

2.4 MCP Copilot Agent

  • The MCP Copilot Agent treats tool use as adaptive decisions under partial observability, with Route, Execute, and Response actions informed by descriptions and execution feedback.Its ReACT loop uses the server–tool hierarchy; the routing score emphasizes joint alignment while allowing server-level priors to dominate when tool matches are weak.Source: §2.4 · p. 4

3 Experiments and Results

The experiments compare 12 frontier models on task success and tool-use behavior, finding wide performance variation and a correlation between active tool composition and success.

3.1 Setup

  • Experiments cover 12 frontier models, use Deepseek-V3 as default evaluator, and report dialogue turns, tools, executions, retrievals, tokens, and task success.Source: §3.1 · p. 4, §3.2 · p. 5, Table 2 · p. 5

3.2 Main Results

  • Most contemporary models achieve only 30%–50% task success, while the Claude series substantially outperforms them.Source: Table 2 · p. 5, §3.2 · p. 5
  • Claude-Sonnet-4 uses 2.71 tools, 5.59 executions, and 2.98 retrievals, while conservative Gemini-2.5-Pro and DeepSeek-V3 use one tool on average.Source: §3.2 · p. 5
  • Claude-Opus-4's 3.40 tools accompany 70.53% success, while Qwen3-32B under-explores at 1.16 tools, showing raw tool counts alone do not determine performance.Source: §3.2 · p. 5
  • Figure 3 relates evaluation performance to human agreement across evaluators on Claude trajectories; Deepseek-V3 reaches 78.95%, versus 60–70% for three lower-agreement evaluators.Source: Figure 3 · p. 5, §3.2 · p. 5
  • Most evaluators agree more with LLM-generated than manual key points, while Deepseek-V3 best leverages human annotations.Source: §3.2 · p. 5, Figure 4 · p. 6

4 Analysis

The analysis examines cost, retrieval settings, evaluator reliability, and agent failure modes. It finds retrieval errors dominant, while tool exploration, evaluation choices, and error handling shape MCP-agent performance and assessment.

4.1 Efficiency Analysis

  • Claude-Sonnet-4 averaged 20.09 steps and 6.70M tokens, versus Qwen3-235B’s 16.76 steps and 3.73M tokens, alongside higher tool use (2.71 versus 1.59 tools per task).The analysis attributes higher token consumption to additional steps and more aggressive tool use, including exploratory trials and error-recovery attempts.Source: §4.1 · p. 6
  • Figure 5 plots log-price against performance and marks a Pareto frontier to identify cost-effective model choices.The scatter plot uses colors to represent model families.Source: §4.1 · p. 6, Figure 5 · p. 6

4.2 Impact of Tool Retrieval Configuration

  • Reducing retrieved tools from 5 to 1 lowered success from 78.95% to 64.21% (p=0.02), while increasing k to 10 left success at 78.95% (p=1.00).Replacing Qwen3-Embedding-0.6B with BGE-M3 [3] changed success from 78.95% to 76.84% (p=0.84), a nonsignificant difference.Source: §4.2 · p. 6

4.3 Single Evaluator Bias Analysis

  • 77.89% majority-vote human agreement was below DeepSeek-v3’s 81.05% but above DeepSeek-r1’s 58.95%; evaluator rankings aligned strongly (Kendall’s τ-b=0.8837, p=0.00008).The reported maximum human agreement across evaluator outputs was 97.89%.Source: §4.3 · pp. 6–7

4.4 Evaluation Error Analysis

  • DeepSeek-v3 evaluation achieved F1 scores of 87.32 for claude-sonnet-4 and 83.08 for claude-opus-4; hallucinated completion accounted for 45% of evaluation errors.Other error categories were output completeness (30%), granularity mismatch (17.5%), and others (7.5%).Source: §4.4 · p. 7

4.5 Error Analysis

  • Query errors reflect poor task decomposition or query refinement; tool errors involve incorrect invocation, while other errors include timeouts and invocation failures.The paper links these categories respectively to planning, contextual precision and memory, and insufficient fault-tolerant recovery mechanisms.Source: §4.5 · p. 8
  • Retrieval errors dominate agent failures, with semantic mismatches and hierarchical retrieval shortcomings limiting discovery of appropriate tools.The examples include a tool supporting audio extraction that retrieval misses for a request to convert a video to MP3.Source: §4.5 · p. 8

5 Related Work

Prior tool-use benchmarks trade real API execution against stability and often remain API-centric. MCP offers a unified basis for broader toolsets, but existing MCP benchmarks and large-scale retrieval approaches still have scale, diversity, or adaptability limits.

5.1 Tool-Use Benchmarks

  • Earlier tool-use benchmarks simulate APIs, collect APIs without executing calls, or integrate real APIs whose changing interfaces can make tools unusable; many remain API-centric and lack local-file functionality.LiveMCPBench uses MCP as a stable, unified interface for a practical toolset intended to address API instability and functional limitations.Source: §5.1 · p. 8

5.2 MCP Benchmarks

  • Existing MCP benchmarks generally cover around 10 servers, while large-toolset retrieval approaches can use rigid pipelines or manually curated toolsets with limited functional diversity.LiveMCPBench introduces a large-scale MCP toolset and evaluates daily-task completion through tool use.Source: §5.2 · p. 8

6 Conclusion

The paper presents a benchmark, tool collection, evaluation framework, and agent for studying tool use in large MCP ecosystems. Experiments identify retrieval and composition as central challenges for future systems.

6 Conclusion

  • LiveMCPBench contains 95 daily tasks, and LiveMCPTool provides 527 tools; LiveMCPEval evaluates dynamic tasks and diverse valid solutions.The paper also builds MCP Copilot Agent to study retrieval and composition in large-scale MCP settings.Source: §6 · p. 8
  • Experiments across 12 frontier models identify retrieval as the primary bottleneck and effective tool composition as equally critical, motivating advances in both capabilities.The conclusion calls for future systems to jointly advance retrieval and robust tool composition.Source: §6 · pp. 8–9

A Limitations

The limitations concern dependence on LLM judgment, assumptions about how trajectories and tool descriptions represent task performance, and maintenance beyond the fixed release.

A Limitations

  • LiveMCPEval relies on LLM-based evaluation, so potential model biases may influence results despite validation through human experiments.The authors analyze judgment failure cases to improve evaluation robustness.Source: A Limitations · p. 10
  • The assessment assumes trajectories and tool descriptions reflect task performance without verifying final environmental impact, which could compromise reliability as the toolset expands.The authors inspect LiveMCPTool quality to reduce discrepancies between tool effects and descriptions.Source: A Limitations · p. 10
  • The work does not address long-term security, versioning, or maintenance of deployed servers; reproducibility is supported by a fixed, validated toolset version.The authors describe long-term sustainability as a separate research direction.Source: A Limitations · p. 10

B Ethical Considerations

The paper discusses both the potential benefits of large-scale tool-use agents and risks from deceptive tools or erroneous invocations. It also reports its LLM use and released reproducibility materials.

B Ethical Considerations

  • Misleading descriptions may disguise harmful tools, while erroneous invocations such as unintended file deletion can cause security or financial losses.The paper underscores the need for safeguards in tool validation and execution monitoring.Source: B Ethical Considerations · p. 10

C LLM Usage

  • The authors used GPT-based tools only for grammar, readability, and wording, while retaining all conceptual, technical, and scientific contributions.Source: C LLM Usage · p. 10

D Reproducibility statement

E.1 Task Statistics

LiveMCPBench covers varied daily tasks and emphasizes domain breadth and compositional complexity. Its tasks were proposed and validated by annotators, and its tool collection was checked for real-time data retrieval.

E.1 Task Statistics

  • The 95 retained tasks prioritize domain coverage, generality, and compositional complexity, including multi-hop and cross-domain scenarios.The benchmark spans six task categories designed to reflect common real-life scenarios.Source: §E.1 · pp. 10–11
  • Annotators used 55 of 70 servers and 150 of 527 tools, while the benchmark captures the noisy or redundant tools that are often unnecessary for completion.Source: §E.1 · p. 11

E.2 Annotation Principles

  • Proposers explored the toolset, created realistic and time-sensitive tasks requiring multiple tools, and documented the tools and key points needed to complete them.Verifiers consolidated similar tasks and assessed feasibility and execution quality.Source: §E.2 · p. 11

F.1 Toolset Statistics

F.2 Tool Verification for LiveMCPTool.

  • For tools requiring online information, verification confirmed retrieval from real online sources rather than simulated responses or static cached datasets.Source: §F.2 · p. 12

G Implementation Details

The paper standardizes model and retrieval settings and reports evaluation metrics alongside analyses of tool-use hallucination. Hallucination-related evaluator disagreements were uncommon, and error distributions remained similar across evaluators.

G Implementation Details

  • Experiments deployed two models, Qwen2.5-72B-Instruct and Qwen3-Embedding-0.6B, and accessed 11 proprietary models through OpenRouter (8), Anthropic Console (2), and Google AI Studio (1).A uniform temperature parameter of 0.7 was used across all experiments.Source: G Implementation Details · p. 12

H.1 Evaluation Metrics and Judge Model

  • Single-run success rate is the primary metric; pass@k measures success in at least one of k samples, while pass^k measures success in all k samples.DeepSeek-V3 is recommended as the judge model for comparable results.Source: §H.1 · pp. 12–13

H.2 Tool-Use Hallucination

  • Hallucinated tool calls or outputs can mislead trajectory-based evaluation, while fixed rules and answer-based checks are difficult to scale in dynamic MCP settings.Source: §H.2 · p. 13
  • Evidence-grounded evaluation decomposes tasks into completion criteria and checks each criterion against raw tool outputs and the tool sequence.The approach targets fabricated intermediate information in multi-step reasoning.Source: §H.2 · p. 13
  • 1.6% was the inconsistency rate between the DeepSeek-V3 evaluator and human judgments due to tool-use hallucination; it detected 8 of 11 such errors (72.7%).Hallucination occurred in 9.00% of 814 Claude-Sonnet-4 tool invocations.Source: §H.2 · p. 13
  • 33.3% Completeness Error, 46.2% Hallucinated Completion, 15.4% Granularity Mismatch, and 5.1% Others were reported with Qwen2.5-72B and DeepSeek-V3 evaluators.The similar error distributions suggest hallucination-related failures are captured consistently and are unlikely to materially distort evaluation results.Source: §H.2 · p. 13

H.3 Case Study: Error Evaluation Examples

LLM-generated key points captured similar critical steps to human annotations despite different ordering. The section also illustrates a long-trajectory evaluation failure and describes how human key points are normalized.

H.3 Case Study: Error Evaluation Examples

  • The evaluator marked a task successful after file creation despite the agent not acquiring correct information; the authors identify lengthy, complex trajectories as a limitation.Dynamic assessment of each trajectory step is proposed, but the authors note it would significantly compromise evaluation efficiency.Source: §H.3 · p. 14

H.4 Human Key Points

  • A key point is an atomic, auditable step tied to a concrete tool or observable signal, with one tool-mediated action per point.Source: §H.4 · p. 14
  • Verifiers normalize key points into imperative, single-action statements with explicit dependencies, deduplicating overlaps while retaining the coarsest still-auditable granularity.Source: §H.4 · p. 14

H.5 Case Study: Human and LLM Key Points Examples

  • Human and LLM-generated key points captured similar critical steps despite ordering differences, supporting LLM-generated key points when human annotations are unavailable.Source: §H.5 · p. 14

I Case Study: Error Examples

The case study classifies agent failures into four error types, with retrieval errors most frequent. The prompts define a routed workflow for finding and executing tools and evaluating task completion.

I Case Study: Error Examples

  • Retrieve errors account for 50.00% of errors, compared with 13.33% for Query Error and 18.33% each for Tool Error and Other Error.Figures 14–17 illustrate the four error types.Source: I Case Study: Error Examples · p. 14, Figure 14 · p. 15, Figure 15 · p. 15
  • Query and Other errors reflect agent-architecture design flaws, Tool errors relate to LLM parameter and description handling, and Retrieve errors reflect retrieval-system limitations.The categories point to distinct sources of difficulty: task completion mechanisms, contextual tool use, and relevant-tool identification.Source: I Case Study: Error Examples · p. 14

J Prompts

  • The agent is instructed to use retrieval and execution tools where possible and provide a complete response in its single reply.The prompt describes retrieval as searching a large toolset and execution as invoking retrieved tools.Source: J Prompts · p. 17
  • The mcp-copilot route tool is the discovery step for finding servers and tools when a task is unfamiliar, uncertain, vague, or complex.Its query includes server and tool descriptions in a tool_assistant tag.Source: J Prompts · p. 17
  • The agent executes a tool only after routing to it, and may retry a failed execution up to three times.Execution specifies a server, tool, and optional parameter dictionary.Source: J Prompts · p. 17
  • The evaluation prompt checks whether task requirements were met using key points and tool-call history, including whether information came from tools and required files were created.Its output format requests a success or failure status.Source: J Prompts · p. 17
  • The key-points prompt extracts critical task elements explicitly stated in the description and excludes unstated inferences.It requests a numbered list of explicit key points.Source: J Prompts · p. 18

Related papers

Loading 2508.01780v2…