PaperlayerView exact revision on arXiv

Source-linked AI summary

MCPSecBench: A Systematic Security Benchmark and Playground for Testing Model Context Protocols

Yixuan Yang, Cuifeng Gao, Daoyuan Wu, Yufan Chen, Yingjiu Li, Shuai Wang

arXiv:2508.13220v3cs.CRcs.AI

TL;DR

MCP broadens LLM attack surfaces as agents connect to external data and tools, creating a need for system-wide security evaluation. The paper formalizes secure MCP and builds MCPSecBench around 17 attacks and four surfaces. Every attack surface compromised at least one evaluated platform, while current protections blocked malicious behavior in only a few scenarios.

  • Problem

    MCP expands LLM attack surfaces by connecting agents to external data, tools, and services, while existing research lacks holistic security evaluation.

  • Method

    The paper formalizes secure MCP, derives four attack surfaces and 17 attack types, and implements MCPSecBench for extensible cross-platform testing.

  • Results

    Every attack surface compromised at least one evaluated platform, with core protocol and host-side vulnerabilities affecting Claude, OpenAI, and Cursor.

  • Takeaways & Limitations

    MCPSecBench provides a systematic platform for rigorous evaluation of MCP threats and defensive strategies across protocol layers.

  • Takeaways & Limitations

    Current protection approaches blocked malicious behavior in only a few scenarios, with an average success rate below 50%.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly integrated into real-world applications via the Model Context Protocol (MCP), a universal open standard for connecting AI agents with data sources and external tools. While MCP enhances the capabilities of LLM-based agents, it also introduces new security risks and significantly expands their attack surface. In this paper, we present the first formalization of a secure MCP and its required specifications. Based on this foundation, we establish a comprehensive MCP security taxonomy that extends existing models by incorporating protocol-level and host-side threats, identifying 17 distinct attack types across four primary attack surfaces. Building on these specifications, we introduce MCPSecBench, a systematic security benchmark and playground that integrates prompt datasets, MCP servers, MCP clients, attack scripts, a GUI test harness, and protection mechanisms to evaluate these threats across three major MCP platforms. MCPSecBench is designed to be modular and extensible, allowing researchers to incorporate custom implementations of clients, servers, and transport protocols for rigorous assessment. Our evaluation across three major MCP platforms reveals that all attack surfaces yield successful compromises. Core vulnerabilities universally affect Claude, OpenAI, and Cursor, while server-side and specific client-side attacks exhibit considerable variability across different hosts and models. Furthermore, current protection mechanisms proved largely ineffective, achieving an average success rate of less than 30%. Overall, MCPSecBench standardizes the evaluation of MCP security and enables rigorous testing across all protocol layers.

1. Introduction

MCP expands LLM capabilities by connecting agents to external data, tools, and services, but broadens the security attack surface. The paper formalizes this landscape, introduces MCPSecBench, and finds successful compromises across all attack surfaces and evaluated platforms.

  • Motivation: MCP connects AI agents to external data, tools, and services while introducing client-, protocol-, server-, and host-side vulnerabilities.These risks matter because MCP-powered agents increasingly interact with sensitive enterprise systems and physical infrastructure.
  • Research gap: Existing MCP security research focuses mainly on isolated threats, lacking a holistic framework for evaluating risks across the architecture.The paper addresses this gap through a formal definition of secure MCP and a taxonomy of attack surfaces and types.
  • Contributions: The paper introduces a formal secure-MCP specification and classifies 17 attack types across four attack surfaces.The taxonomy covers client, protocol, server, and host surfaces.
  • Contributions: MCPSecBench integrates prompt data, MCP clients and servers, attack scripts, and a GUI harness for extensible security evaluation.Researchers can evaluate custom MCP hosts, clients, servers, and transport protocols within the playground.
  • Evaluation: Every identified attack surface successfully compromised at least one evaluated platform, while core protocol and host-side vulnerabilities affected Claude, OpenAI, and Cursor.Platform susceptibility varied for other vectors, including prompt injection, which had 0% success on Claude Desktop and 100% on Cursor.

2. MCP Background

MCP is an open client-server standard that connects AI assistants with external data and services. Its workflow discovers tools, mediates user-driven tool calls, and returns server results, with transports and features supporting flexible access and security boundaries.

  • MCP architecture: MCP is a universal open standard for connecting AI assistants to external data and services.It standardizes access across diverse data sources and tools for real-world applications.
  • MCP architecture: MCP hosts contain clients that maintain isolated one-to-one connections with individual servers and manage requests, sessions, and responses.The host orchestrates connections and task execution while clients act as intermediaries.
  • MCP protocol: MCP communication uses JSON-RPC requests, responses, and notifications over stdio or streamable HTTP, with optional server-sent events for server responses.Stdio is common for local and CLI integrations, while streamable HTTP supports client-to-server communication.
  • MCP servers: MCP servers expose tools, resources, and prompts, supported by metadata and configuration.Tools invoke external services, resources provide contextual data access, and prompts standardize recurring operations.
  • MCP workflow: The MCP workflow proceeds through tool discovery, user interaction, and tool execution.Servers provide tool lists, clients register them for the LLM, and execution results are relayed back through the host to the user.
  • MCP features: Sampling, Roots, and Elicitation extend MCP with completion requests, resource restrictions, and dynamic workflows.Roots enforce operational boundaries and least privilege, while Sampling supports multi-step workflows and human review.

3. Formalization of Secure MCP

The paper formalizes secure MCP by defining system components, data flows, and behavioral constraints whose violations constitute potential attacks. These specifications establish client, protocol, server, and host attack surfaces for benchmark design.

  • Formalization objective: The paper formalizes secure MCP to characterize theoretical risks and define security requirements for its major components.Any violation of these specifications is treated as a potential MCP attack.
  • System model: The MCP system is modeled as a 5-tuple containing clients, protocol, servers, data objects, and sequential execution flows.The data objects include initializations, prompts, messages, and responses.
  • System model: The formal execution flow maps prompts and initializations through clients and the protocol to received messages and server responses.The notation distinguishes received messages from sent messages to account for possible network alterations.
  • Client side: Client-side security requires accepted prompts to satisfy global security invariants and produce tool calls appropriate to user intent.The constraint targets forbidden or unintended behaviors generated from user prompts.
  • Protocol side: Protocol-side security requires received messages to match messages sent by authenticated peers without in-transit tampering.This specification addresses message integrity and man-in-the-middle alteration.
  • Server side: Server-side security requires executed tools to be trusted and their outputs to remain benign relative to the initial prompt.The specification connects tool verification, execution, response causality, and safety constraints.
  • Host side: Host-side security requires authorized operations and a correctly configured environment, extending the threat model to implementation flaws and misconfigurations.Side-effecting operations are distinguished from passive reads through the host specification.
  • From specifications to attack surfaces: The four formal specifications define the client, protocol, server, and host attack surfaces used to implement MCPSECBENCH.The specifications serve both as correctness rules and as a blueprint for potential attack vectors.

4. MCPSECBENCH

MCPSECBENCH is a systematic security benchmark and playground that evaluates MCP threats across client-, protocol-, server-, and host-side surfaces using attacks, protections, and an automated workflow. It combines vulnerable reference components, adversarial prompts, and a GUI harness to assess specification violations during MCP operations.

  • The benchmark targets four attack surfaces—client-side, protocol-side, server-side, and host-side—alongside runtime protection mechanisms.
  • Typical Workflow: The typical workflow connects hosts, clients, and servers through MCP before relaying tool outputs from servers to users.
  • Specifications: The benchmark checks host configuration and command-injection resistance, safe client prompt parsing, and protocol data integrity and confidentiality.
  • Attack Component: 17 distinct attacks formalize specification violations across the benchmark’s evaluated MCP components.
  • Protection Component: The benchmark evaluates dynamic protection mechanisms, including MCIP-Guardian and Firewall, during MCP interaction.
  • Evaluation Component: Attacks succeed when they cause specification violations, fail when hosts or AI proactively refuse them, and are separately categorized when unrelated technical issues stop execution.
  • It combines five reference MCP servers, intentionally vulnerable clients, major-provider-compatible hosts, two protections, adversarial prompts, and a GUI test harness.

5. Evaluation

MCPSECBENCH evaluates 17 attacks across three platforms using ASR and RR, revealing universal protocol-side compromise, high server- and host-side vulnerability, and uneven protection effectiveness. MCIP and FAN mitigate selected attacks, but protection performance varies substantially by attack vector and host, with MCIP imposing higher costs than FAN.

  • Evaluation Setup: 17 attacks were evaluated across Claude Desktop, OpenAI, and Cursor, with each attack tested 15 times; defenses were assessed on 11 attacks.The platform evaluation used ASR and RR, while the defense evaluation focused on attacks for which at least one mechanism showed detection capability.
  • Results by Attack Surface: Client-side attacks had the lowest overall ASR and highest RR; Claude Desktop recorded 33.3% ASR.These attacks primarily used prompt injection and “Confused AI” tactics, which the passage describes as increasingly detectable by modern LLMs.
  • Results by Attack Surface: 100% ASR occurred for protocol-side attacks across Claude, OpenAI, and Cursor.The result indicates that protocol-level exploits succeeded universally in the evaluated conditions.
  • Results by Attack Surface: Host-side ASRs were 58.3%, 75%, and 81.7% for Claude, OpenAI, and Cursor, respectively, while server-side ASRs all exceeded 75%.Server-side attacks exploited hidden tool misuse that bypassed linguistic filters associated with model refusals.
  • Results by Attack Type: Schema Inconsistencies, Vulnerable Client, MCP Rebinding, and Man-in-the-Middle each achieved 100% ASR across all platforms, and evaluated defenses did not mitigate them.These structural attacks remained successful in controlled environments despite the protection mechanisms.
  • Protection Mechanisms: MCIP and FAN strongly mitigated Indirect Prompt Injection and Vulnerable Server attacks, with refusal rates frequently exceeding 80%, but their effectiveness diverged on Sandbox Escape and across hosts.FAN mitigated 77.8% of Sandbox Escape attacks in certain configurations, whereas MCIP achieved 100% RR against Data Exfiltration on Claude Desktop but had minimal impact on Cursor and OpenAI.
  • Cost Analysis: MCIP approximately doubled costs for Claude Desktop and OpenAI and reached nearly four times baseline cost on Cursor, while FAN remained comparable to the native environment.Reported MCIP costs were $0.41 to $0.76 for Claude Desktop, $0.11 to $0.18 for OpenAI, and $0.46 versus $0.11 for Cursor.

6. Related Work

Prior MCP-security research exposed important risks but generally covered narrow attack scopes, isolated surfaces, or proprietary hosts. MCPSecBench addresses these limitations with a formalization-grounded benchmark spanning all four attack surfaces and 17 vulnerability types.

  • Prior MCP-security studies mainly focus on server-side vectors and proprietary hosts, limiting reproducibility and evaluation scope.
  • Most prior studies address fewer than three vulnerability types and remain restricted to a single attack surface, except MCIP’s ten types.
  • Existing defense evaluations are highly fragmented, ranging from authors’ own mechanisms to isolated safeguards.
  • MCPSecBench is grounded in a comprehensive MCP formalization and covers client, protocol, server, and host attack surfaces.
  • The framework expands evaluation to 17 vulnerability types and supports universal protection mechanisms for systematic defense assessment.

7. Conclusion

The paper formalizes MCP security across four attack surfaces and builds MCPSecBench to test 17 attack vectors. Experiments show that vulnerabilities across any MCP component can enable sensitive-data exfiltration or host compromise despite existing defenses.

  • The paper formalizes MCP security specifications and identifies four critical attack surfaces.
  • MCPSecBench integrates prompts, servers, clients, attack scripts, and defenses to simulate 17 distinct attack vectors.
  • Experiments show that vulnerabilities across any MCP component can enable sensitive-data exfiltration or host-environment compromise.
  • The benchmark is intended to support future analysis of novel threats and rigorous evaluation of defensive strategies.

Impact Statement

The study characterizes MCP threats across client, protocol, server, and host components, including attacks that can bypass authorization or alter intended tool behavior. It emphasizes defensive evaluation in controlled environments and responsible deployment.

  • The research systematically characterizes MCP threats and evaluates attack vectors and protection mechanisms across three major providers.
  • The work’s stated objective is defensive, supporting vulnerability assessment and stronger security controls through controlled, isolated experiments.
  • Prompt injection and confused-AI attacks can cause tool selection or execution to diverge from the user’s intended task.
  • Protocol attacks include rebinding and man-in-the-middle scenarios that expose servers or compromise message integrity and confidentiality.
  • Server- and host-side attacks include poisoned tools, vulnerable servers, configuration drift, and sandbox escapes that can produce unauthorized operations.

B. Components of MCPSECBENCH

MCPSECBENCH combines multi-surface security criteria with modular components for prompts, hosts, malicious servers, transport attacks, GUI automation, and protections. Its design supports automated, extensible testing of MCP systems.

  • The benchmark evaluates client, protocol, server, and host behavior using criteria tailored to correct tool use, secure transmission, benign execution, and safe task completion.
  • Its GUI test harness identifies input vectors, captures responses, manages authorization prompts, and uses an LLM judge to assess attack success.
  • The prompt dataset maps carefully designed prompts to taxonomy attacks, enabling systematic triggering of client- and server-side vulnerabilities.
  • MCP endpoints implement Claude, OpenAI, and Cursor hosts and test schema inconsistencies, slash-command overlap, and CVE-2025-6514.
  • Malicious MCP servers cover metadata, prompts, tools, resources, and configuration to exercise server-side attack types.
  • The transport module exposes risks from unencrypted or unauthenticated communication through man-in-the-middle and DNS rebinding attacks.
  • Protection mechanisms include code scanning, judge agents, and behavior rules, spanning static and runtime defenses.

C. Extension

MCPSECBENCH supports evaluation and extension across MCP servers, clients, providers, protections, and connection methods. Its modular design lets researchers add custom attack vectors, protection strategies, and components.

  • MCPSECBENCH evaluates MCP servers, clients, providers, and protection mechanisms across 17 attack vectors.
  • The benchmark covers 17 attacks across four attack surfaces and can incorporate custom attack vectors or protection strategies.
  • MCP Servers Extension: The platform supports local, HTTP, and SSE connections for MCP servers.Remote configurations use URLs ending in /mcp for HTTP and /sse for SSE.
  • MCP Clients Extension: Custom MCP clients can connect simultaneously to multiple servers and providers using standard local or remote connection methods.
  • MCP Providers Extension: MCPSECBENCH supports three major MCP providers and can be adapted to OpenAI API-compatible or proprietary provider APIs.Proprietary integrations require implementing a single chat method.
  • MCP Protection Extension: The framework includes two protection mechanisms and allows additional protections to be integrated as MCP servers.

D. Case Study

The case study tests diverse protocol, host, client, and server-side attacks across MCP platforms. Results show widespread susceptibility, with defenses varying substantially by attack and host.

  • Prompt Injection: Claude Desktop rejected prompt injection with 0% ASR and 100% RR, while OpenAI refused 73.3% and Cursor refused 0%.
  • Tool/Service Misuse via “Confused AI”: Tool/service misuse produced 40–73.3% ASR for OpenAI and Cursor, whereas Claude Desktop robustly defended against the deceptive tool definitions.The attack relies on randomness, so ASR and RR varied across runs.
  • Schema Inconsistencies: Schema inconsistencies can terminate MCP host processes and render systems unavailable.The reported causes include type mismatches, missing required keys, and syntax errors.
  • Man-in-the-Middle: The three MCP hosts lacked encryption and authentication sufficient to prevent interception, replay, and modification of MCP traffic.
  • Tool Shadowing Attack: OpenAI showed 53.3% ASR for tool shadowing, Claude Desktop 20%, and Cursor 6.7%.
  • 100% ASR occurred for prompt-directed data exfiltration, package name squatting, tool poisoning, and slash command overlap in the reported tests.Data exfiltration reached 100% ASR when the tool-list disclosure was explicitly requested.
  • Configuration Drift: All three hosts lacked authentication against configuration drift exposing a local server at 0.0.0.0.Any user on the same network could connect and execute actions within the server environment.
  • Sandbox Escape: OpenAI and Cursor reached 100% ASR for sandbox escape, while Claude Desktop reached 0% ASR.

E. Cost Evaluation

The cost evaluation reports token consumption and associated prices for each attack test. It highlights higher resource use for complex deceptive prompts and distinguishes applicable from non-applicable defenses or attacks.

  • The cost evaluation records token consumption and associated prices for each attack vector.
  • Tool/service misuse via “Confused AI” consumed substantial resources in the No Defense baseline because it required complex prompts.
  • FAN had a token footprint comparable to the No Defense baseline because it primarily used static rule-setting.
  • N/A indicates that a defense or attack was not applicable to the tested attack or MCP platform.
Loading 2508.13220v3…