PaperlayerView exact revision on arXiv

Source-linked AI summary

MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models

Xuanqi Gao, Siyi Xie, Juan Zhai, Shiqing Ma, Chao Shen

arXiv:2505.16700v2cs.AI

TL;DR

Existing evaluations do not adequately assess LLM tool use in the MCP paradigm, motivating a benchmark that covers distinct task types and uses objective measures. MCP-RADAR's evaluation finds a recurring gap between making tool calls and selecting and using tools effectively, alongside differences in model performance and efficiency.

  • Problem

    Existing evaluations provide limited insight into tool-use capabilities and can fail to distinguish tool-based problem solving from recalled knowledge or capture real-world interaction complexity.

    Source: §1 · p. 2

  • Method

    MCP-RADAR evaluates Precise Answer and Fuzzy Match tasks using answer and operation matching, across real and high-fidelity mock MCP tools.

    Source: §1 · p. 2

  • Results

    Models often make tool calls but struggle with semantic tool selection, multi-step planning, and information synthesis; the closed-source math advantage narrows to less than 10% in web search.

    Source: §1 · p. 2, §6.2 · p. 9

  • Takeaways & Limitations

    The findings point to tool selection, multi-step planning, and information synthesis as directions for model and tool development.

    Source: §6.2 · p. 9

    The benchmark measures performance but does not explicitly address or mitigate societal biases that evaluated models may reflect.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

As Large Language Models (LLMs) evolve from passive text generators to active reasoning agents capable of interacting with external tools, the Model Context Protocol (MCP) has emerged as a key standardized framework for dynamic tool discovery and orchestration. Despite its widespread industry adoption, existing evaluation methods do not adequately assess tool utilization capabilities under this new paradigm. To address this gap, this paper introduces MCP-RADAR, the first comprehensive benchmark specifically designed to evaluate LLM performance within the MCP framework. MCP-RADAR features a challenging dataset of 507 tasks spanning six domains: mathematical reasoning, web search, email, calendar, file management, and terminal operations. It quantifies performance based on two primary criteria: answer correctness and operational accuracy. To closely emulate real-world usage, our evaluation employs both authentic MCP tools and high-fidelity simulations of official tools. Unlike traditional benchmarks that rely on subjective human evaluation or binary success metrics, MCP-RADAR adopts objective, quantifiable measurements across multiple task domains, including computational resource efficiency and the number of successful tool-invocation rounds. Our evaluation of leading closed-source and open-source LLMs reveals distinct capability profiles and highlights a significant trade-off between accuracy and efficiency. Our findings provide actionable insights for both LLM developers and tool creators, establishing a standardized methodology applicable to the broader LLM agent ecosystem. All implementations, configurations, and datasets are publicly available at https://anonymous.4open.science/r/MCPRadar-B143.

1 INTRODUCTION

MCP-RADAR addresses gaps in tool-use evaluation with a benchmark spanning two task types and six domains, assessed through accuracy, efficiency, and resource-use measures. Its evaluation finds distinct model capability profiles and recurring errors in functional tool selection.

1 INTRODUCTION

  • Existing evaluations offer limited insight into tool use, including whether models solve problems with tools or merely recall knowledge and whether simulations capture real interactions.The introduction argues that rigorous, standardized evaluation within the MCP paradigm is important.Source: §1 · p. 2
  • MCP-RADAR combines Precise Answer tasks with single correct answers and Fuzzy Match tasks requiring correct operation sequences across six domains.The domains are mathematical reasoning, web search, email, calendar, file operations, and terminal.Source: §1 · p. 2
  • The evaluation uses answer matching and operation matching to measure answer accuracy, tool-selection efficiency, and computational resource efficiency.It evaluates ten leading open- and closed-source LLMs using real-world MCP tools and high-fidelity mock tools.Source: §1 · p. 2
  • Closed-source models outperform open-source counterparts in mathematical reasoning, while the web-search gap narrows to less than 10%; semantically plausible but incorrect tool choices recur.The authors describe this pattern as a basis for recommendations for LLM development and MCP tool design.Source: §1 · p. 2

2 RELATED WORK

Related work traces the move toward standardized tool interaction and identifies limitations in existing tool-use benchmarks. The paper positions MCP-RADAR as an evaluation of LLM performance in varied MCP scenarios.

2.1 THE EVOLUTION TOWARDS STANDARDIZED TOOL USE

  • MCP is presented as a unified interaction standard addressing tool-invocation challenges after API-based integrations created expansion and maintenance burdens without a common context-delivery mechanism.The passage describes MCP as formally introduced and open-sourced in that context.Source: §2.1 · p. 3

2.2 EVALUATING TOOL AND MCP PROFICIENCY

  • Prior evaluations leave gaps in efficiency and protocol interaction, complex multi-turn or multi-tool scenarios, and real-world diversity; MCP-RADAR targets varied MCP usage scenarios.The cited limitations include operational-only and entirely synthetic datasets.Source: §2.2 · p. 3

3 MCP-RADAR DATA GENERATION

MCP-RADAR contains 507 instances across six domains, divided into tasks with definitive answers and tasks requiring tool operations. Its curation uses adapted benchmark problems for precise answers and controlled, programmatically generated scenarios for fuzzy matches.

3 MCP-RADAR DATA GENERATION

  • The benchmark comprises 507 instances across six real-world domains, organized as Precise Answer and Fuzzy Match tasks.Precise Answer tasks require a single ground-truth value; Fuzzy Match tasks require a tool invocation with appropriate parameters.Source: §3 · p. 3

3.1 PRECISE ANSWER DATA CURATION

  • Precise Answer curation adapts authoritative math and web-search datasets, filters queries solvable without tools, annotates valid tools, and evaluates the final answer.Verified open-source calculator and web-search MCP tools are integrated for execution.Source: §3.1 · p. 4

3.2 FUZZY MATCH DATA CURATION

  • Fuzzy Match generation uses controlled tool environments and programmatic templates to create query-and-action pairs for single- and multi-tool scenarios.Email and calendar mocks use local databases rather than live operations, while file-management and terminal tasks use community-developed tools.Source: §3.2 · pp. 4–5
  • Multi-tool tasks combine templates for the three most frequently used tools in each domain, with a strictly defined invocation sequence to ensure a unique, verifiable solution.The authors constrain tool pairings to avoid the combinatorial explosion of exhaustive combinations.Source: §3.2 · p. 5

4 EXPERIMENT

The experiment evaluates ten models using 49 MCP tools across six domains, with a limit of ten interaction rounds per task. Its metrics distinguish final-answer correctness from operational-call success and resource cost.

4.1 EXPERIMENTAL SETUP

  • Ten open- and closed-source models solve tasks using 49 MCP tools across six domains, with a maximum of K=10 interaction rounds before task failure.Models receive a system message, the user question, and the available tools; output requirements differ by task type.Source: §4.1 · p. 5

4.2 EVALUATION METRICS

  • Precise Answer tasks measure final-answer accuracy and computational resource efficiency, without scoring intermediate tool-invocation sequences or penalizing alternative valid tool paths.The final extracted answer is matched exactly to ground truth; resource cost includes token consumption.Source: §4.2 · p. 6
  • Fuzzy Match tasks score the decisive tool name and parameters, successful invocations per interaction turn, and computational resource efficiency.Dialogue Turn Success Rate counts invocations with an applicable tool and correctly formatted parameters; resource values are max-min normalized for cross-model comparison.Source: §4.2 · p. 6

5 RESULTS

MCP-RADAR results show that tool-call execution can outpace semantically appropriate task solving, while performance and resource use vary across models and tasks. Accuracy generally improves with more interaction rounds but gains plateau for most models at K ≥10.

5.1 MAIN RESULTS

  • Websearch average success was 20.7% for closed-source models versus 10.8% for open-source models, with success rates below 30% for all models.The passage attributes the difficulty to needing both the correct tool and a precise query.Source: §5.1 · p. 6
  • GPT-4o’s Filemanagement DTSR of 84.5% exceeded its ACC of 43%, a 40.7% gap tied to syntactically correct calls but inappropriate tool selection.The table compares performance metrics across four MCP-RADAR domains.Source: §5.1 · pp. 6–7, Table 3 · p. 7
  • Hints about the number of required tools yielded only 2.5% to 5% improvement, pointing to tool choice and parameterization as the harder capability boundary.The experiment tested whether indicating the required tool count would improve multi-tool task performance.Source: §5.1 · p. 7, Table 4 · p. 7
  • Leading closed-source models showed robust, well-rounded capabilities, while some open-source models reached competitive accuracy with higher token consumption.Gemini-2.5-Pro stood out across diverse domains; Qwen balanced accuracy and resource efficiency among open-source models.Source: §5.1 · p. 7, Figure 4 · p. 8

5.2 ABLATION STUDY

  • As the maximum allowed rounds K increased, overall task accuracy improved for most models, but improvement slowed considerably at K ≥10 and eventually began to plateau.K = 10 was selected as the standard setting for all main experiments to balance solution accuracy and reasonable interaction latency.Source: §5.2 · p. 7

6 ANALYSIS AND DISCUSSION

The error analysis groups failures into tool-use, reasoning, and information-synthesis errors, with dominant failure types differing between precise-answer tasks and fuzzy-match or complex-parameter tasks. The discussion identifies a gap between executing calls and using tools effectively, and offers recommendations for model training and tool design.

6.1 ERROR ANALYSIS

  • The analysis distinguishes Tool-Use Errors, Reasoning Errors, and Information Synthesis Errors, including invalid arguments, omitted or redundant calls, and failures to integrate or extract tool results.Tool-use errors also include choosing a tool inappropriate to the task; reasoning errors include faulty conclusions despite accurate tool output.Source: §6.1 · pp. 8–9
  • Precise-answer tasks most often failed through faulty reasoning and tool omission, while fuzzy-match or complex-parameter tasks more often had parameter errors and inappropriate tool invocation.Niche web queries and math tasks featured tool omission; tools with many arguments increased the risk of incorrect data entry.Source: §6.1 · p. 9

6.2 DISCUSSION AND CONCLUSION

  • Across the findings, models often execute calls but struggle with precise tool selection, multi-step planning, and synthesizing tool outputs into effective solutions.The paper presents these challenges as directions for model and tool development.Source: §6.2 · p. 9
  • For LLM development, the paper recommends improving recognition of when external tools are needed and decomposing complex problems into tool-solvable steps.The authors note that models often default to parametric knowledge and attempt complex problems, including advanced mathematics, with one tool call.Source: §6.2 · pp. 9–10
  • For MCP tool developers, the paper recommends concise, precise descriptions and granular, single-purpose tools that can be combined for complex tasks.The authors report that verbose or ambiguous descriptions increase cognitive load and invocation errors.Source: §6.2 · p. 10

ETHICS STATEMENT

The ethics statement frames MCP-RADAR as an objective tool-use benchmark intended to support responsible agent development. It acknowledges dual-use risk and limitations in bias mitigation while describing safeguards for operational evaluations.

ETHICS STATEMENT

  • More capable tool-using agents could be repurposed for malicious activity, while identifying weaknesses such as incorrect tool selection may inform safer systems.The work presents objective evaluation metrics as a way to foster transparency and guide responsible development; it does not propose a new model architecture.Source: ETHICS STATEMENT · p. 10
  • Operational email, calendar, and terminal tasks ran in a semi-sandboxed environment using custom mock servers that replicate tool functionality without executing real operations.The evaluated LLMs may reflect training-data biases, which the benchmark does not explicitly address or mitigate.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement says the benchmark implementation, configurations, evaluation scripts, and dataset are publicly available. It also points to details on dataset construction, metrics, prompts, and tools.

REPRODUCIBILITY STATEMENT

  • The complete MCP-RADAR implementation, configurations, evaluation scripts, and full dataset are publicly available.The statement provides a repository link for access.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The data-generation account distinguishes repurposed public datasets for Precise Answer tasks from template-based programmatic generation for Fuzzy Match tasks.The statement directs readers to the corresponding sections and appendix examples.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The benchmark combines open-source MCP servers with custom-built mock tools designed to replicate official specifications.The statement says experimental prompts, maximum interaction rounds, and tool implementation details are provided for replication.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 DATASET DETAILS

The appendix gives examples of Precise Answer queries and Fuzzy Match templates, then describes a comparison of concise and ReAct prompts. The supplied text identifies the comparison setup but not its results.

A.1 DATASET DETAILS

  • A Precise Answer example asks how many stops lie between South Station and Windsor Gardens on the MBTA Franklin-Foxboro line, with ground-truth answer 10.The example is introduced as a precise-answer query.Source: §A.1 · p. 16
  • Fuzzy Match examples use single-tool templates for email, calendar, file-management, and terminal tasks.The examples include emailing someone and marking a message, adding an item to a schedule, returning file contents, and starting Python scripts.Source: §A.1 · p. 16

A.2 SYSTEM PROMPT

  • The appendix contrasts concise and ReAct prompt formats; the ReAct math prompt requires a Thought–Action–Observation loop and calculator-tool calls.The concise prompt also instructs the model to use calculator tools rather than answer directly.Source: §A.2 · pp. 16–17
  • The prompt comparison tested ten randomly selected datasets from each domain using two prompts, with results assigned to Table 5.The supplied table caption names the prompt-effect comparison but gives no outcome values.Source: §A.2 · p. 17, Table 5 · p. 17

B RELATED EXPERIMENT RESULTS

The related experiments include worked examples, a noted redundant-search behavior, and a table scoped to dialogue rounds and average accuracy. The following sections show experimental formatting examples and disclose the roles of LLMs in dataset construction and manuscript preparation.

B RELATED EXPERIMENT RESULTS

  • The worked polynomial example ends with the two solutions 2x + 1 and -2x - 1.The accompanying reasoning says the factorization implies one of two factors is identically zero.Source: B RELATED EXPERIMENT RESULTS · p. 18
  • The authors flag redundant tool invocation as a failure pattern: the model repeatedly queries the same website address.A web-search example asks for USGS locations of a nonnative fish and shows the assistant identifying the clown anemonefish and invoking search.Source: B RELATED EXPERIMENT RESULTS · p. 18
  • Table 6 is scoped to average accuracy across domains as the number of dialogue rounds K varies.The supplied caption does not report the comparison's results.Source: Table 6 · p. 19

C EXPERIMENTAL PROCEDURE FORMATTING

  • The experimental-formatting examples show a FireCrawl MCP server with a web-search tool that accepts a query, plus an example record listing tool usage and token counts.The server configuration includes an npx command and an API-key environment variable.Source: C EXPERIMENTAL PROCEDURE FORMATTING · pp. 19–20

D THE USE OF LARGE LANGUAGE MODELS

  • LLMs assisted with selecting Precise Answer queries and polishing manuscript language, while the authors say core scientific contributions and final analysis were performed by humans.Fuzzy Match question-answer pairs were generated programmatically from author-designed templates.Source: D THE USE OF LARGE LANGUAGE MODELS · p. 20

Related papers

Loading 2505.16700v2…