Source-linked AI summary
LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries
Ming Yin, Dinghan Shen, Silei Xu, Sixun Dong, Mian Zhang, Yebowen Hu, Shujian Liu, Jianbing Han, Simin Ma, Song Wang, Sathish Reddy Indurthi, Xun Wang, Yiran Chen, Kaiqiang Song
TL;DR
Existing benchmarks provide limited coverage of realistic, dynamic tasks that require coordinating diverse MCP tools. LiveMCP-101 introduces 101 such tasks and evaluates agents against real-time reference outputs generated by a parallel agent following validated plans. Across 18 models, even frontier LLMs achieve task success rates below 60%.
Problem
Existing benchmarks primarily address single-step calls, static or synthetic environments, or narrow tool sets, limiting assessment of agents coordinating dynamically discovered MCP tools across real-time backends.
Source: §1 · p. 1Method
LiveMCP-101 pairs 101 real-world multi-step MCP tasks with validated execution plans and evaluates agents in parallel against reference outputs produced by a plan-following agent.
Source: §1 · p. 2Results
Even frontier LLMs achieve a task success rate below 60% across 18 models on real-world multi-step MCP tasks.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Tool calling has emerged as a critical capability for AI agents. In contrast to conventional tool calling frameworks that rely on static, provider-specific tool definitions, the Model Context Protocol (MCP) offers a unified interface to discover and invoke tools dynamically. However, there is a significant gap in benchmarking multi-step tasks using diverse MCP tools in realistic, dynamic scenarios. In this work, we present LiveMCP-101, a benchmark of 101 real-world queries that require coordinated use of multiple MCP tools. To address temporal variability in real-world tool responses, we introduce a parallel evaluation framework where a reference agent executes a validated plan simultaneously to produce real-time reference outputs. Experiments show that even frontier LLMs achieve a success rate below 60\%, highlighting challenges in multi-step tool use. Comprehensive error analysis identifies seven failure modes spanning tool planning, parameterization, and output handling, pointing to concrete directions for improving current models. LiveMCP-101 sets a rigorous standard for evaluating real-world agent capabilities, advancing toward autonomous agent systems that reliably execute complex tasks through MCP tool orchestration.
1. Introduction
LiveMCP-101 addresses the shortage of benchmarks for realistic, dynamic, multi-step MCP tool use with 101 real-world tasks and a parallel evaluation protocol anchored in validated plans. Across 18 models, even frontier LLMs achieve task success rates below 60%, and trajectory analysis identifies seven failure modes.
1. Introduction
- Existing benchmarks often target single-step calls, static or synthetic settings, or narrow tool sets, leaving dynamic multi-tool MCP use inadequately assessed.Real-world queries may also require multi-step reasoning to handle nuanced context and strict constraints.Source: §1 · p. 1
- The benchmark contains 101 diverse real-world tasks requiring coordinated MCP tool use, evaluated in parallel against reference outputs from validated plans.The plans specify tool sequences, parameterization, and post-processing; the reference and evaluated agents run simultaneously to address time-varying responses.Source: §1 · p. 2
- Even frontier LLMs achieve a task success rate below 60% across 18 models on realistic, multi-step MCP tasks.The results expose a substantial gap between current agent capabilities and the demands of real-world task execution.Source: §1 · p. 2
- Trajectory analysis identifies seven failure modes across tool planning and orchestration, parameterization, and output handling.The analysis is intended to inform targeted improvements to agent capability in dynamic, multi-step MCP orchestration.Source: §1 · p. 2
2. Related Work
Prior tool-calling benchmarks commonly use static tool definitions or environments, while early MCP benchmarks remain limited in scope. LiveMCP-101 distinguishes itself with challenging real-world tasks and objectively verifiable ground truth from validated plans.
2. Related Work
- Traditional tool-calling approaches rely on manually defined, statically injected schemas, while existing benchmarks often use static definitions and mock environments.These settings limit evaluation of dynamic tool discovery and adaptation to changing tool outputs.Source: §2 · p. 2
- MCP standardizes tool discovery and interaction across servers, including real-time data sources whose responses can vary over time.Agents can query and handshake with available tools and resources at runtime without provider-specific configurations.Source: §2 · p. 2
- Earlier MCP benchmarks cover fewer servers or use static settings, while concurrent live evaluations lack objectively verifiable ground truth.LiveMCP-101 uses human-validated execution plans for real-time references and includes three difficulty tiers, with tasks requiring an average of 5.4 tool-calling steps.Source: §2 · pp. 2–3, Table 1 · p. 3
3. LiveMCP-101
LiveMCP-101 constructs curated, tiered queries and validated execution plans for dynamic MCP tasks, then evaluates agents against real-time references and judge-scored results and trajectories.
3.1. Construction
- Queries are generated from diverse MCP domains and human-refined for clarity, balanced difficulty, solvability, and objective verifiability.Difficulty is based on reasoning complexity and tool-chain length; Easy tasks use straightforward short chains, while Hard tasks require multi-constraint reasoning and extensive orchestration.Source: §3.1 · p. 3
- The benchmark covers 101 tasks across three difficulty tiers, with validated plans averaging 5.4 steps and reaching 15 steps.The tiers contain 30 Easy, 30 Medium, and 41 Hard tasks.Source: §3.1 · p. 3, Figure 3 · p. 4
- Each query receives an explicitly parameterized execution plan and output-processing code, iteratively revised and manually checked across at least three runs.Plan revision required approximately 120 expert-hours; finalized plans yield time-aligned results when followed in the live environment.Source: §3.1 · p. 4
3.2.1. EVALUATION FRAMEWORK
- For each task, a reference agent follows the validated plan in parallel to produce an evaluation-time reference output for the live environment.This addresses nonstationary tool responses, for which fixed ground-truth answers are unreliable at test time.Source: §3.1 · p. 3, §3.2.1 · p. 4
3.2.2. EVALUATION METRICS
- An LLM judge scores final results and execution trajectories on a 1–5 Likert scale by checking them against the reference outputs.The scores are normalized for reporting.Source: §3.2.2 · p. 4
- Reported metrics include task success rate, average result and trajectory scores, average output-token consumption, and average tool calls.Result scores reflect solution quality, while trajectory scores assess logical coherence, completeness, and correctness.Source: §3.2.2 · p. 4
4. Experiments
LiveMCP-101 evaluates 18 models on coordinated MCP tasks and finds that GPT-5 leads overall, yet performance declines with difficulty. Ablations examine iteration budgets and server-pool breadth, while human comparisons assess judge reliability.
4.1. Experimental Setup
- The evaluation covers 18 representative models using ReAct, a 30-round limit, per-task pools of 15 servers and 76–125 tools, and five performance metrics.The metrics are TSR, ARS, ATS, average output tokens, and average tool calls.Source: §4.1 · pp. 4–5
4.2. Main Results
- GPT-5 achieves the best overall performance, leading across all difficulty tiers, but even the strongest model reaches only 39.02% TSR on Hard.Rankings by TSR and ARS are broadly consistent; performance degrades with task difficulty across models.Source: §4.2 · p. 5, Table 2 · p. 6
4.3. Ablation Study
- Raising the iteration limit from 15 to 25 rounds consistently improves TSR, but gains largely plateau beyond 25.The authors attribute continued gains through 25 rounds to additional tool exploration and error recovery; beyond that, model capability constrains performance.Source: §4.3 · p. 6, Figure 5 · p. 7
- As server pools grow, weaker and mid-tier models often decline, while top-tier systems remain largely stable.The study varies pools from 6 to 15 servers; expanded search space can increase selection overhead and spurious tool use.Source: §4.3 · p. 6, Figure 5 · p. 7
4.4. Analysis of LLM-as-a-Judge
- Human–LLM agreement exceeds 0.85 for result evaluations and 0.78 for trajectory evaluations across six models.Across three LLM judges and all 101 tasks, the paper also reports high correlation, attributing consistency to objectively verifiable answers.Source: §4.4 · p. 7, Figure 6 · p. 7
5. Discussion
Token-efficiency patterns differ between closed- and open-source models, and execution-log analysis identifies seven failure subtypes across planning, parameterization, and output handling. Semantic parameter errors are especially common, while output parsing remains a persistent last-mile problem.
5.1. Token Efficiency
- Closed-source models’ TSR rises rapidly at small token budgets, then plateaus; open-source models achieve substantially lower TSR at comparable or higher budgets.The paper associates early-token gains with planning, tool probing, and constraint checks, while later tokens add redundancy rather than new evidence.Source: §5.1 · p. 7
5.2. Failure Analysis
- The failure taxonomy groups seven subtypes under tool planning and orchestration, parameterization, and output handling.The subtypes include missed requirements, self-solving without needed tools, unproductive thinking, wrong tool selection, syntactic and semantic parameter errors, and output parsing errors.Source: §5.2 · p. 8, Figure 7 · p. 8
- Semantic errors dominate: top-performing models show 16–25% rates, while lower-performing models exceed 40%, including GPT-4.1-mini.These errors involve well-formed parameters that do not match task intent, such as mis-scoped queries or incorrect identifiers.Source: §5.2 · p. 8
- Syntactic errors are negligible for frontier models but reach 48% for Llama-3.3-70B-Instruct; limited MCP-specific training is offered as a plausible cause.The paper suggests targeted fine-tuning on MCP tool-call schemas could reduce these errors.Source: §5.2 · p. 8
- Overconfident self-solving is more common in mid-tier models, while output parsing remains a pervasive last-mile failure with structured data such as JSON.Parsing failures occur when a tool returns a correct result but the agent fails to extract or transform it accurately.Source: §5.2 · p. 8
6. Conclusion
The conclusion presents LiveMCP-101 as a benchmark and parallel evaluation protocol for dynamic, multi-step MCP tool use. Across 18 models, frontier LLMs remain below 60% task success, and analysis identifies seven failure modes.
6. Conclusion
- LiveMCP-101 pairs 101 real-world multi-step tasks with validated-plan parallel evaluation to mitigate temporal drift and score outputs and execution trajectories.Across 18 models, even frontier LLMs remain below 60% task success; the analysis identifies seven failure modes spanning planning, parameterization, and output handling.Source: §6 · p. 8
7. Impact Statement
The impact statement frames the benchmark as a stress-testing framework intended to support more reliable agent deployment. The appendix illustrates validated execution plans and diagnoses an example of an agent missing task requirements.
7. Impact Statement
- The impact statement presents rigorous stress-testing as a way to assess agent robustness before real-world application and help mitigate deployment risks.It notes potential societal risks associated with widespread autonomous-agent deployment.Source: §7 · p. 9
A. Example Execution Plans
- Easy, Medium, and Hard example plans specify MCP tools, concrete parameters, and purposes; the reference agent follows them to generate deterministic real-time scoring outputs.The examples are shown in Figures 8, 9, and 10.Source: Figure 8 · p. 12, Figure 9 · p. 12, Figure 10 · p. 13
B. Failure Cases
- Failure cases are organized as task queries, agent trajectories, and diagnoses, with the trajectory showing both reasoning steps and tool calls.The presented reasoning is lightly abridged for readability.Source: B. Failure Cases · p. 14
B.1. Ignoring requirement
- In the LangChain example, the agent stops after minimal GitHub queries and returns non-conforming JSON, omitting project analysis and contributor identifiers.The task requested analysis of recent commits, the contributor’s top starred repositories, and identifying information.Source: §B.1 · pp. 14–15
B.2. Overconfident self-solving
The examples distinguish failures to use the needed tool from choosing a tool that returns the wrong data. They show these errors can leave requested results incorrect or absent, even when an output file is written.
B.2. Overconfident self-solving
- The agent returned incorrect equation roots without calling wolfram-alpha, exemplifying overconfident self-solving.The expected tool call was absent, and the reported roots were incorrect.Source: §B.2 · p. 15
B.3. Unproductive thinking
- The agent planned a PPP analysis but never retrieved market rates, computed valuations, or wrote the requested JSON file.The missing execution included exchange-rate, code-interpreter, and filesystem tool calls.Source: §B.3 · p. 16
B.4. Wrong tool selection
- The agent used a general weather tool instead of a forecast tool, so the report lacked tomorrow’s 10:00 AM data despite being written.The returned report had no tomorrow filter or 10:00 AM granularity.Source: §B.4 · pp. 16–17
B.5. Syntactic errors
The movie-report example shows how parameter-type errors can derail a multi-step task, while later steps may still write a deliverable. The agent also treated a popular-movie list as the week’s trending list without support in the passages.
B.5. Syntactic errors
- The agent assumed the first five popular movies represented the week’s trending movies, although the request asked for trending titles.The planned sequence was to retrieve movie details and YouTube trailers before creating the report.Source: §B.5 · p. 17
- The YouTube search call had invalid arguments, but the agent subsequently wrote the requested report file.The passages report a successful file write, not that the report contained valid movie details or trailer links.Source: §B.5 · pp. 17–18
- A string page value failed integer validation, and a placeholder movie ID then failed validation, preventing valid movie retrieval.The initial error propagated to subsequent calls with invalid movie IDs.Source: §B.5 · pp. 17–18
B.6. Semantic errors
The retailer-riddle example illustrates semantic errors: the agent misidentified a retailer and proceeded with searches and a file based on that mistaken interpretation. The file write succeeded, but the results were misaligned with the task.
B.6. Semantic errors
- The agent used an unverified Golden State Warriors and Chase Center assumption as its workplace anchor for nearby-store searches.The searches returned nearby candidates relative to that assumed anchor, and place details were fetched for three retailers.Source: §B.6 · pp. 18–19
- The requested file was written successfully, although the retailer misidentification left the results misaligned with the task.A successful write did not correct the semantic error in the retailer selection.Source: §B.6 · p. 19
- The arena-named retailer was Target, but the agent resolved it as Chase, a misidentification that propagated to incorrect tool parameters and misaligned results.The agent searched for Chase Bank rather than the intended retailer.Source: §B.6 · pp. 18–19
B.7. Output parsing errors
The video-analysis example separates successful retrieval and computation from errors in handling tool outputs. A value was miscomputed in the spreadsheet after the agent retrieved video details and calculated views per minute.
B.7. Output parsing errors
- The agent retrieved five video IDs and their views and durations, then used Python to convert durations and compute engagement.The requested measure was views divided by video duration in minutes.Source: §B.7 · pp. 19–20
- The workbook and worksheet were created, and the data write completed despite the incorrect video #2 value.The passages report successful workbook and sheet creation before the miswritten engagement value.Source: §B.7 · p. 20
- 81,999.9999 was written for video #2’s engagement, instead of the expected 81,833.3333, due to an output-processing error.The engagement rate is described as views per minute.Source: §B.7 · p. 20
C. Additional Experimental Results
The additional experiments report reproducibility across independent runs and robustness across LLM judges. The accompanying prompts specify how to score task results and tool trajectories, and how the evaluated and reference agents should act.
C. Additional Experimental Results
- Low standard deviation across runs indicates high reproducibility, while low standard deviation across three LLM judges demonstrates evaluation-framework robustness.Table 3 covers three independent runs; Table 4 reports task success rate and average result score for GPT-4.1, Claude-4-Sonnet, and Gemini-2.5-Pro.Source: C. Additional Experimental Results · p. 21, Table 3 · p. 21, Table 4 · p. 21
D.1. Result Evaluation Prompt
- Result evaluation compares the agent’s answer with the reference against the query, using a 1–5 correctness rubric that accepts minor short-term fluctuations in variable values.Formatting and wording differences are acceptable, but material differences are not excused as dynamic data or timing.Source: §D.1 · p. 22
D.2. Trajectory Evaluation Prompt
- Trajectory evaluation scores tool-chain logic, completeness, and efficiency on a 1–5 scale, while allowing valid approaches that differ from the reference chain.The rubric ranges from optimal or clearly valid trajectories to illogical or largely incorrect ones.Source: §D.2 · p. 22
D.3. ReAct Prompt
- The ReAct prompt directs the agent to think, call a tool, observe its result, and repeat until the task is complete without requesting guidance.Source: §D.3 · p. 23
D.4. Reference agent prompt
- The reference agent must follow the supplied execution plan in order, use its specified tools without adding or skipping steps, and verify each result before proceeding.Source: §D.4 · p. 23