PaperlayerView exact revision on arXiv

Source-linked AI summary

MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation

Wenhao Wang, Peizhi Niu, Gongyi Zou, Xiyuan Yang, Jingxing Wang, Haoting Shi, Yaxin Du, Jingyi Chai, Xianghe Pang, Shuo Tang, Yanfeng Wang, Siheng Chen

arXiv:2606.02470v1cs.AI

TL;DR

Existing benchmarks overlook the challenges of personalized MCP tools tied to individual accounts and histories. MCP-Persona introduces a simulation-based benchmark for such tools and finds that even state-of-the-art agents struggle with personalized tool use, especially when necessary information is implicit or requires coordination.

  • Problem

    Existing benchmarks focus on generic tool orchestration while underexploring agents operating tools tied to individual accounts, preferences, and histories.

  • Method

    MCP-Persona combines Tool-Traverse simulation with 173 human-verified tasks across four personalized application scenarios.

  • Results

    Even state-of-the-art agents struggle with personalized tool use, including implicit grounding, multi-step state maintenance, and cross-tool coordination.

  • Takeaways & Limitations

    MCP-Persona provides a reproducible, privacy-preserving testbed for evaluating and improving personalization-aware agents.

  • Takeaways & Limitations

    Personalized tools may amplify societal biases and enable discriminatory outcomes, over-reliance, targeted phishing, social engineering, or automated spam.

Abstract

from arXiv · show

The Model Context Protocol (MCP) has emerged as a transformative standard for connecting large language models (LLMs) with external data sources and tools, and has been rapidly adopted across personal applications and development platforms. However, existing benchmarks predominantly focus on generic information-seeking tools and fail to capture the practical challenges posed by personal social applications, where tools interact with individual accounts or local databases. To bridge this critical gap, we introduce MCP-Persona, the first benchmark specifically designed for evaluating agent performance on real-world, personalized MCP tools. MCP-Persona encompasses a diverse set of widely-used applications, ranging from social media platforms like Reddit and Xiaohongshu (Rednote) to enterprise collaboration suites such as Lark (Feishu) and Slack. Our extensive experiments on various state-of-the-art (SOTA) agents demonstrate their significant struggles with personalized tool use, thereby highlighting the benchmark's crucial role in identifying and addressing these limitations. MCP-Persona is publicly available at https://github.com/wwh0411/MCP-Persona}{https://github.com/wwh0411/MCP-Persona.

1. Introduction

MCP-Persona addresses the underexplored evaluation of agents on personalized MCP tools by simulating private, stateful application environments while preserving privacy and fairness. It introduces Tool-Traverse, Context-Tree, and Persona-Gen, and evaluates diverse agents across 173 human-verified tasks in four personalized scenarios.

  • Motivation: Existing tool-use and MCP research largely overlooks personalized tools tightly coupled with individual user contexts.Evaluation remains underexplored because personalized tools require private data, costly setup, privacy-preserving benchmark construction, and stable executable environments.
  • Benchmark: MCP-Persona evaluates tool-enhanced agents on personalized applications through the interaction of tools, user contexts, and tasks.The platform targets real-world applications including Slack, Rednote, and Instagram while aiming to preserve user privacy and evaluation fairness.
  • Methods: Tool-Traverse simulates real-world personal MCP tools by traversing authentic function calls collected from sandboxed servers and accounts.Self-FC expands human-curated seed function calls to improve coverage and diversity before simulation.
  • Methods: Context-Tree models application contexts as structured hierarchies and supports stateful operations while replacing sensitive fields with fake values.The method prioritizes authentic online content when available and supplies populated context instances to simulated tools.
  • Methods: Persona-Gen converts tool-chain prototypes into realistic tasks through automated synthesis, context injection, intentional obfuscation, and manual refinement.Prototype generation is constrained by dependency, diversity, and realism.
  • Findings: 173 human-verified tasks span four representative personalized scenarios, and experiments find that even GPT-5 struggles with some unseen-tool tasks.Agents have difficulty acquiring information embedded in the environment when it is not explicitly stated in the instruction.

2. Related Work

Personalized agents increasingly support individualized, localized, and everyday needs through customizable skills and on-device mobile experiences. However, existing benchmarks rarely evaluate account-associated, permission-sensitive tools in social media and enterprise collaboration, motivating MCP-Persona’s simulated real-world personalized-tool benchmark.

  • Personalized Agents: Personalized agents have proliferated into widely deployed products supporting individualized, localized, and everyday user needs.The Skills framework provides modular, extensible user-specific capabilities that developers and end users can customize for localized environments and real-world demands.
  • Personalized Agents: Doubao Phone delivers personalized mobile experiences for communication, shopping, and food delivery through on-device intelligence.Users can interact with agents that support these daily activities.
  • Limitations of Existing Benchmarks: Existing research rarely focuses on real-world personalized scenarios involving social media platforms and enterprise collaboration systems.Most existing tool-use benchmarks and datasets primarily target generic search-oriented tools, tasks, and simplified scenarios.
  • Limitations of Existing Benchmarks: ToolAthlon does not evaluate social media or enterprise collaboration tasks because account-associated and permission-sensitive environments are intrinsically difficult.These platforms are among the most widely used applications in everyday scenarios.
  • MCP-Persona: MCP-Persona simulates tools from popular applications such as Instagram and Lark as the first benchmark explicitly designed for personalized tool usage in social communication scenarios.The benchmark is introduced to address the gap in evaluating individualized demands in real-world agent usage.

3. Methodology

MCP-Persona builds personalized MCP simulations by combining curated server collection, systematic tool traversal, executable code synthesis, mutable context trees, and manually reviewed task generation. The methodology captures both successful and failed tool behavior while preserving stateful, relational, and personalized interactions.

  • MCP Server and Tool Collection: The benchmark uses manual curation and automated discovery to collect personalized MCP applications requiring authentication and state management.The collection emphasizes high-value applications across categories including enterprise collaboration platforms.
  • Function Call Pool Construction: Tool Traversal probes live servers using valid seed calls and adversarial perturbations to capture successful behaviors, failure modes, decision boundaries, and error handling.Adversarial inputs cover type mismatches, schema violations, boundary conditions, and semantic conflicts, with only verified calls added to the function-call pool.
  • Code-as-Simulation: LLMs synthesize executable Python simulation kernels from tool schemas, behavioral traces, and context-handler APIs instead of relying on manual rules or static mock responses.The kernels validate inputs, check entity existence, reproduce identified failure modes, and run in a sandbox environment.
  • Dynamic Context Construction: A dynamic context handler and user-rooted context tree support entity loading, saving, querying, modification, persistence across turns, and relational linking.Typed entities are organized through ownership and containment relations, with identifier-indexed child maps enabling lookup and updates.
  • Personalized Task Synthesis: Personalized tasks are synthesized by transforming tool-invocation and documentation-based instruction skeletons through context injection, decoupling, and fuzzification, followed by manual review.Context content uses enumerated, free-form, random, or sanitized authentic generation methods before cross-entity linking.

4. Experiments

Experiments show that current LLM agents remain unreliable on personalized MCP tasks, especially under cross-server coordination, rich context, and long dependency chains. Tool-Traverse improves simulation fidelity, while skill documentation and restricted tool candidates improve agent performance.

  • Single-Server Breakdown: Email tasks are easiest, while social media, collaboration, and content-management tools are harder because they require entity resolution, cross-user interactions, or long-document editing.Content-management tools perform worst among the described single-server task families.
  • Cross-Server Breakdown: Hodgepodge cross-server tasks are consistently the most difficult because they involve more cross-server interactions and complex dependency chains.The benchmark evaluates Lark-Centric, Rednote-Centric, and Hodgepodge scenarios.
  • Simulation Fidelity: Tool-Traverse substantially improves simulated Lark-tool fidelity over Vanilla, which achieves only 53.3% F1 and produces generic errors rather than specific response structures.Tool-Traverse roughly triples both TF-IDF and METEOR response-similarity scores and better handles valid complex inputs.
  • Ablations: Adding detailed skill descriptions tends to improve agent performance, while restricting candidate tools to ground-truth servers is especially beneficial for longer contexts.The comparisons cover Lark and Rednote subsets and contrast all available tools with ground-truth-server tools.
  • Failure Analysis: Failure traces reveal recurring errors from insufficient environment exploration, skipped dependency resolution, and context overload that degrades constraint adherence and recall.Agents may hallucinate recipients or IDs, omit latent schema-required steps, and fail after progressive context stacking or voluminous tool outputs.

5. Conclusion

MCP-Persona is a benchmark for evaluating tool-augmented agents in realistic personalized settings, built from simulated MCP servers and human-verified tasks. Experiments show that even SOTA agents struggle with personalized tool use, especially implicit grounding, multi-step state maintenance, and crosstool coordination.

  • Benchmark: MCP-Persona evaluates tool-augmented agents in realistic personalized settings across 12 simulated MCP servers and 173 human-verified tasks.The benchmark spans a wide range of personal applications that previous work has struggled to handle.
  • Findings: Experiments reveal that even SOTA agents fall short on personalized tool use.The benchmark exposes limitations that remain in current state-of-the-art agents.
  • Findings: The main weaknesses involve implicit grounding, multi-step state maintenance, and crosstool coordination.These challenges characterize personalized tool use in the benchmark’s realistic application settings.

Impact Statement

The paper warns that personalized-tool benchmarks may create societal harms, including amplified bias and inequitable outcomes in high-stakes domains. It therefore calls for future work to pair task-performance evaluation with privacy, transparency, and accountability metrics.

  • Societal risks: Personalized tools may perpetuate or amplify societal biases when trained on historical data reflecting systemic inequities.The paper identifies potential consequences in finance, healthcare, and legal applications, including discriminatory loan, diagnosis, or case-preparation outcomes.
  • Benchmark incentives: Benchmarks emphasizing efficiency or task success may incentivize powerful but inequitable models.The concern is that optimization around performance metrics alone can overlook fairness-related harms.
  • Mitigation: Future research should integrate rigorous metrics for privacy, transparency, and accountability to mitigate potential harms.The recommendation is explicitly framed as a necessary complement to performance evaluation.

A. Implementation Details · A.1. Tool and Data Details

The benchmark documents its MCP servers and organizes them by application role to support reproducibility and future research. These servers provide models with external tools and data for retrieval, content management, and multi-step information gathering.

  • A.1. Tool and Data Details: MCP-Persona includes named MCP servers with links to support reproducibility.The benchmark provides server identities and official repository URLs.
  • A.1. Tool and Data Details: The servers are grouped into generic search, social communication, enterprise collaboration, local note-taking, and email categories.This categorization reflects the range of personal application contexts represented in the benchmark.
  • A.1. Tool and Data Details: Official repository URLs accompany the MCP server listings.These links facilitate access to the server implementations and future research use.
  • A.1. Tool and Data Details: The MCP servers expose external tools and data sources to the model.Their interfaces enable interaction with information and application data beyond the model itself.
  • A.1. Tool and Data Details: The exposed capabilities support retrieval, content management, and multi-step information gathering.These activities summarize the operational tasks enabled by the benchmark’s server environment.
  • A.1. Tool and Data Details: Table 8 lists the MCP servers included in the MCP-Persona Benchmark.The table presents the benchmark’s server inventory.

A.2. Experiment Details

The experiments evaluate multi-modal tool-calling across diverse MCP servers using behavioral alignment and response-similarity metrics. Ablations examine skill quality, tool-candidate selection, and context distractors, including their effects on personalized task performance.

  • Main Experiments: 173 multi-modal tool-calling tasks span MCP servers including Lark, Xiaohongshu, Slack, and Notion, with agents operating for up to 20 rounds.Tasks provide user instructions, available tools, and necessary and unnecessary context across information-seeking, personal file, and personal chat categories.
  • Tool Simulation Fidelity: Tool simulation fidelity is assessed by replaying recorded inputs and using an LLM judge to determine whether simulated responses substantively fulfill tool declarations.The judge receives the tool name, description, input schema, and full response body, avoiding reliance on unreliable boolean status flags.
  • Evaluation Metrics: Behavioral Alignment compares binary success/failure outcomes using Accuracy, Precision, Recall, and F1, while Response Similarity evaluates canonical JSON responses with TF-IDF and ROUGE.TF-IDF probes server-specific error codes and identifier formats through low-frequency-token sensitivity.
  • Skills Ablation: The skills ablation compares Vanilla/No Skills, + OpenClaw Skills, and + Our Skills on Lark and Xiaohongshu/Rednote subsets.The subsets contain 63 Lark tasks and 64 Xiaohongshu/Rednote tasks; the refined guides cover Lark calendar, contact, and IM workflows and Xiaohongshu task workflows.
  • Tool Candidate Ablation: The candidate-tool ablation compares All Tools with Selected Tools, retaining only ground-truth servers and required tools in the latter setting.All Tools exposes 140 tools across MCP servers, including 78 personalized tools for personal chat and file management; server relevance is identified from task labels and ground-truth tool-calling chains.
  • Context Confusion Ablation: The context-confusion ablation compares Only Necessary Context with With Context Distractors by adding five randomly sampled paragraphs while holding the instruction and necessary context unchanged.The distractors come from an entity pool of approximately 16k English DBpedia paragraphs, and the study finds performance sensitive to distractor information.

A.3. Model Details

Table 10 lists the LLM versions used in the evaluation, covering proprietary and open-weight systems from major providers. This model set supports broad comparisons across model families and deployment sources.

  • Model versions: Table 10 presents the versions of the LLMs used in the evaluation.The table specifies the model versions included in the experiments.
  • Model coverage: The evaluation includes both proprietary and open-weight systems from major providers.This selection spans different provider and access models.
  • Comparison scope: The model set enables broad comparisons across different model families and deployment sources.The coverage is designed to compare heterogeneous systems and origins.

B. Examples and Prompts

This section presents realistic, challenging personalized tasks and the annotation and prompt-generation procedures used to make them executable, evaluable, and reproducible. It specifies alignment checks, checkpoint-based ground truth, dynamic context CRUD handling, and simulation-kernel validation requirements.

  • B.1. Task Examples: Persona-Gen produces realistic, detail-rich, challenging yet executable tasks grounded in authentic workflows across diverse application domains.The tasks reflect the complexity and ambiguity of everyday personalized tool use.
  • B.2.1. TASK ANNOTATION GUIDE: Task annotation enforces semantic consistency among instructions, decoupled context, and ground-truth tool chains so personalized tasks remain coherent, solvable, and unambiguous.Annotators prioritize clear logical chains, refine instructions for real usage, minimize derivable context, and remove misaligned tasks.
  • B.2.2. CHECKPOINT ANNOTATION GUIDE Execution-Based-GT Annotation Human Guide: Execution-based ground truth represents tasks as checkpoints covering non-personalized searches, personalized searches, state changes, and fallback log-based evaluation.Personalized search checkpoints require exact values, while state changes record the tool, operation type, summary, and affected context ID.
  • B.2.2. CHECKPOINT ANNOTATION GUIDE Execution-Based-GT Annotation Human Guide: Ground-truth annotations must be accurate and reproducible, follow tool-chain sequence, and remain consistent with the environment context.These constraints connect expected outputs and state changes to the execution environment used for evaluation.
  • B.3.1. DYNAMIC CONTEXT HANDLER GENERATION Dynamic Context Handler Generation Prompt: The dynamic context handler prompt requires executable Python CRUD support over Dict[str, Dict[str, Any]] contexts, including loading, saving, retrieval, listing, creation, updating, and deletion.It also requires parent validation, wildcard traversal, and careful bracket-first path parsing so identifiers containing dots are handled correctly.
  • B.3.1. DYNAMIC CONTEXT HANDLER GENERATION Dynamic Context Handler Generation Prompt: The handler prompt specifies bracket-first selector extraction before splitting on dots outside brackets, with wildcard filtering and structure-aware identifier generation.Examples include UUID-like event IDs and email-like calendar IDs, while path IDs act as strict filters for specific paths.
  • B.3.2. PROMPT TEMPLATE FOR SIMULATION KERNEL GENERATION Simulation Kernel Generation Prompt: The simulation-kernel prompt generates Python functions that emulate real API tools over single-user or multi-user dynamic contexts and return structured success, error, and result fields.The kernel loads and selects context, persists every modification, validates inputs and references, and checks entity or parent existence before operations.
  • B.3.2. PROMPT TEMPLATE FOR SIMULATION KERNEL GENERATION Simulation Kernel Generation Prompt: Simulation kernels must validate list, create, update, and delete operations, resolve special context keywords, match expected JSON structures, and emit specific errors as executable code.The prompt requires reference checks for IDs such as calendars and folders, alongside code-only output without markdown formatting.
Loading 2606.02470v1…