Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance

Introduction

• Current benchmarks assume tools respond correctly and on time, leaving resilience to realistic MCP failures untested.Agent quality depends on the MCP channel, server implementations, schemas, and tool-ecosystem health.Source: §1 · p. 1
• MCP-Injector is a drop-in middleware with permanent, transient, and intermittent faults calibrated from 57,240 threads across 1,774 public MCP repositories.It uses seedable schedules and applies without modifying the agent or tools.Source: §1 · p. 2
• A 1,350-cell study found that the larger tested backbone retained no larger share of its fault-free score under calibrated faults, while agent stacks differed in fault response.The paper describes the evidence that framework-level error handling matters more than model scale as suggestive.Source: §1 · p. 2

Background and Definitions

• The MCP tool-call boundary is the structured invocation request and its result or error; prompts, resources, and streaming are out of scope.The experiments and injector operate strictly at this boundary under a non-adversarial threat model.Source: §2 · p. 2
• A service failure is unexpected tool-call behavior, while a fault is its underlying cause; the injector introduces external faults at the agent-server interface.Examples include an error instead of a result, excessive latency, or a schema-violating result.Source: §2 · p. 2
• Timing failures delay or withhold results, while content failures return wrong or unusable results; faults may be permanent, transient, or intermittent.Permanent faults recur until corrective action, transient faults resolve without intervention, and intermittent faults alternate in correlated bursts.Source: §2 · p. 2

3 MCP-Injector

• MCP-Injector is drop-in, reproducible through seeded schedules, low-interference when disabled, and stack-agnostic.The middleware requires no modifications to the agent runtime, MCP client, or servers.Source: §3 · p. 2

3.1 Architecture

• Each invocation passes through Observe, Decide, Act, and Record stages, from reading call context through operator selection and logging.Act forwards the call and may mutate the response; Decide uses the fault profile, persistence state, and seeded randomness.Source: Figure 1 · p. 3

3.2 Fault API

• The fault API separates what failure to emulate, how long it persists, and where it applies.Operators can short-circuit requests, delay forwarding, or mutate response content.Source: §3.2 · p. 3
• Permanent schedules fire on subsequent matching calls, transient schedules fire once and reset, and intermittent schedules hold faults across calls probabilistically.Injection choices are deterministic given the seed and tool-call history, and fired faults are logged; end-to-end runs remain stochastic.Source: §3.2 · p. 3

4 Fault Profile Methodology

• The six failure modes distinguish content errors from timing problems, and together with three persistence regimes define the calibration target space.Content modes include minor mistakes, incorrect or invalid outputs, and explicit errors; timing modes include delayed responses and silent drops.Source: §4 · p. 4
• Researchers assembled 1,774 active public MCP tool-server repositories and produced 103,166 consolidated issue and pull-request threads.They combined public registries, deduplicated repositories, and mined recent GitHub artifacts.Source: §4 · p. 4
• Filtering left 57,240 English-language classified threads after removing duplicates, build-related threads, and non-English threads.The filtering used keyword heuristics and automated and manual passes.Source: §4 · p. 4
• A language model classified threads into failure-mode and persistence pairs, with two researchers independently annotating a stratified sample and a third adjudicating disagreements.Agreement was κ = 0.93 for failure mode and κ = 0.80 for persistence.Source: §4 · p. 4
• The profile aggregates classified incidents into a versioned, seedable distribution, with permanent failures predominating and explicit errors the largest failure share.EXPLICIT_ERROR accounts for 10,393 of 14,829 classified failures.Source: §4 · p. 4, Table 2 · p. 5
• The profile matches relative frequencies of mode-persistence cells among mined failure reports, not operational failure rates.It is 91.3% permanent, and GitHub issues may over-represent persistent, reproducible failures, so the authors treat it as a stress profile.Source: §4 · p. 5

5.1 Benchmark and Task Set

• The benchmark contains 25 single-server tasks generated from successfully connected MCP servers and presented as fuzzy user prompts.Distraction servers are disabled to isolate tool-call faults from conversational noise.Source: §5.1 · p. 5

5.2 Experimental Factors

• The full-factorial experiment crosses two backbones, three agent stacks, three proxy conditions, 25 tasks, and three seeds for 1,350 cells.Task prompts, schemas, context budget, and stopping policy are held constant across stacks, while orchestration behavior varies.Source: §5.2 · pp. 5–6

5.3 Execution and Run Policy

• All reported cells use a 3,000-second task timeout and stack-specific loop bounds; timeouts, exhausted bounds, and unhandled tool exceptions count as system-level failures.Completed runs are scored five times by an LLM judge with randomized rubric order, and median scores are reported.Source: §5.3 · p. 6

5.4 Metrics

• The evaluation tracks rubric-based task quality, system-level failures, and latency decomposed into end-to-end, agent-internal, and evaluation time.Latency supports proxy-intrusiveness analysis under OP0 and fault-amplification analysis under OPP.Source: §5.4 · p. 6

6.1 Calibrated Faults Cause Severe, Pervasive Quality Degradation

• 60%: pooled task-completion score falls from 8.08 to 3.23 under OPP, with every metric dropping significantly and a 57% task-completion drop among cells completed in both conditions.Failures are scored zero in the primary analysis; the completed-only result shows degradation is not solely due to non-completion.Source: §6.1 · p. 6
• Partial degradation is the most common OPP outcome, affecting 56% of cells overall and 44 to 71% across stack and model; the other listed failure categories together account for 21 to 39%.Figure 3 classifies all 450 OPP cells by behavioral outcome.Source: §6.1 · p. 7, Figure 3 · p. 7
• Under OPP, 27 completed runs meet the tool-bypass definition, and ungrounded fulfillment credit makes the task-fulfillment drop 58% rather than 63% under the grounded measure.Seventeen bypass runs made no tool call; the grounded measure requires tool appropriateness of at least 2.Source: §6.1 · p. 7

6.2 Agent Stacks Shape Fault Response

• Stack resilience profiles differ: LangChain has the highest OPP completion rates and median score ratios, while MCPBench and OpenAI show steeper degradation than LangChain in the interaction analysis.The framework effect is suggestive rather than established: intervals approach zero, no multiplicity correction is applied, and MCPBench receives more fault events per cell.Source: §6.2 · pp. 7–8, Figure 4 · p. 8
• After tool failures, MCPBench more often switches tools on the next round, whereas LangChain and OpenAI more often exit the tool loop than retry.The reported behaviors are consistent with the stacks' different planning and ReAct-style loops.Source: §6.2 · p. 8

6.3 Scale Does Not Purchase Fault Tolerance

• The 120B has no detectable resilience-ratio advantage over the 20B (model-size coefficient −0.009, p = 0.96), and the interval cannot exclude a modest effect in either direction.The resilience model scores non-completions as zero; the two backbones have similar fault-free completed-run scores.Source: §6.3 · p. 8
• Under OPP, the 120B completes fewer tasks than the 20B (91.1% vs. 95.6%; difference 4.4 points, 95% CI [−0.2, 9.3]), a descriptive gap rather than an established scale effect.Fault-attributable completion drops are −5.8 points for the 120B and −3.6 for the 20B; their 2.2-point difference is not distinguishable from zero.Source: §6.3 · p. 9
• Greater 120B fault exposure is consistent with the completion gap but does not establish it; degradation also varies sharply by task, from −97% on game_trends to −19% on unit_converter.The paper notes shared context limits and tool-call volume as alternative explanations.Source: §6.3 · p. 9

Construct validity

• The stochastic LLM judge and rubric's reward for some ungrounded answers limit construct validity, while the GitHub-derived fault profile reflects reporting bias and is treated as a stress profile.Content operators also omit schema drift, payload truncation, and partial result omission.Source: §7 · p. 9

Internal validity

• Fault exposure and error handling remain partially entangled because stacks differ in both tool-call volume and error handling, and zero-dose cells plus three seeds limit power for per-task comparisons and rare interactions.Longer tool-call chains amplifying exposure is described as a hypothesis, not an established mechanism.Source: §7 · p. 9

External validity

• External validity is bounded by two backbones from one model family under one serving setup, easy benchmark tasks, and 25 single-server tasks that do not cover multi-server or longer-horizon settings.The study is powered for a permanent-heavy profile; transient and intermittent events were too few to characterize those regimes.Source: §7 · p. 9

8 Broader impacts

• MCP-Injector enables reproducible pre-deployment stress testing against a fault profile calibrated to observed public MCP incidents.The profile is shaped by reporting bias in public GitHub artifacts, so it is a stress test rather than a certification instrument.Source: §8 · p. 9
• An attacker already positioned on the MCP path could repurpose the middleware’s operator primitives.These include fabricated errors, delays, silent drops, and perturbed payloads.Source: §8 · p. 9
• Mitigations include OP0 as the default, fault headers for cooperating loggers, and documentation scoping use to staging environments.The fault header lets cooperating loggers distinguish synthetic from real faults.Source: §8 · pp. 9–10

9 Related work

• Existing tool-use benchmarks [16] [30] [31] [27] [19] [28] assume tools respond correctly and on time; ReliabilityBench [10] is closest but lacks MCP-Injector’s calibrated persistence profile.Xiong et al. [29] study parameter-filling failures inside the LLM, while Agent-SafetyBench [33] and Ruan et al. [23] address adversarial threats.Source: §9 · p. 10
• Unlike infrastructure-level or inter-agent fault injection [22] [13], MCP-Injector injects semantic content, timing, and schema faults at the MCP tool-call interface.These faults exercise agent-level interpretation and recovery logic.Source: §9 · p. 10
• MCP-Injector provides controlled, reproducible MCP failures, unlike post hoc failure catalogues [11] [12] or self-correction methods such as Reflexion [25] that assume reliable tools.Source: §9 · p. 10

10 Conclusion

• 60% drop in task-completion score under the permanent-heavy profile; framework-level error handling appears to matter more than model scale, while the 120B retains no larger share of its fault-free score.Partial degradation, not catastrophic failure, was the most common outcome (56% of cells); stack-by-condition evidence is suggestive.Source: §10 · p. 10

A Fault Profile Calibration Details

• The 18-entry profile combines six failure operators with three persistence types; selection weights follow mined counts, while schedule and content parameters include fixed hyperparameters.UNCERTAIN labels are excluded, and NON_FAILURE is represented by the null-injection OP0 baseline.Source: A Fault Profile Calibration Details · pp. 13–14
• Selection weights equal each injectable operator-persistence cell’s count divided by 13,824, preserving its share of the mined distribution.For example, EXPLICIT_ERROR/PERMANENT weighs 0.6565, while MINOR_MISTAKE/INTERMITTENT weighs 0.0011.Source: A Fault Profile Calibration Details · p. 13
• INTERMITTENT activation probabilities are derived from mined recurring-failure ratios, but the implemented held-entry schedule fires less often than those ratios imply.With d = 0.3, an entry lasts about 4.3 calls on average; its activation probability governs firing during that held period.Source: A Fault Profile Calibration Details · p. 13
• TRANSIENT entries fire once with activation probability 1.0, whereas PERMANENT entries fire on every subsequent matching call.Source: A Fault Profile Calibration Details · pp. 13–14
• The shared INTERMITTENT deactivation value d = 0.3 sets a short average faulty burst, about 3.3 calls in the two-state design.In the shipped injector, a held entry lasts about 4.3 calls on average, short relative to task execution loops.Source: A Fault Profile Calibration Details · p. 14
• Fixed operator settings distinguish disruption types: OP5 delays calls 30 seconds, OP1 sparsely perturbs words, and OP2 corrupts roughly one third of responses.OP3, OP4, and OP6 require no tunable parameters; the remaining fixed values separate semantic intensities and keep delay below the 60-second transport timeout.Source: A Fault Profile Calibration Details · p. 14

B Proxy Non-Intrusiveness: Full Analysis

• No proxy-off versus OP0 score comparison reached significance after correction across 18 tests, and score deltas showed no consistent direction across stacks.The largest absolute mean difference was 0.55 points; a slight proxy effect in mcpbench/20b could not be excluded.Source: B Proxy Non-Intrusiveness: Full Analysis · p. 15
• 0.1 to 0.5 s per tool invocation is the estimated proxy overhead, versus typical LLM inference latency of 2 to 15 s per turn.The reported end-to-end latency deltas varied by stack and included run-to-run inference-time variance.Source: B Proxy Non-Intrusiveness: Full Analysis · p. 15
• OP0 is a valid fault-free baseline, so subsequent OP0-versus-OPP comparisons are not confounded by instrumentation artifacts.Source: B Proxy Non-Intrusiveness: Full Analysis · p. 15

• Figure 5 reports execution-time deltas (OP0 −proxy-off) per task, by stack; their distributions are centered near zero with no systematic positive bias, confirming minimal proxy overhead.Source: Figure 5 · p. 16
• The per-event mix differed from selection weights: permanent faults made up 99.5% of fired events versus 91.3% of weight, while intermittent entries fired only 6 times (0.3%).OP2/PERMANENT rose from 12.5% specified to 28.9% realized; permanent draws fire on every subsequent tool call, so entries and stacks with longer sequences contribute more events.Source: C Injection Fidelity: Full Analysis · p. 16, Figure 6 · p. 17
• Zero-dose cells occurred when agents made no tool call or an intermittent draw never fired; 29 cells made no tool call and 26 had an unfired intermittent draw.Including zero-dose cells in the primary analysis dilutes the measured fault effect; excluding them changes task-completion degradation from −60% to −67%.Source: C Injection Fidelity: Full Analysis · p. 16
• Fault exposure was associated with lower task-completion scores: cells with fired faults scored 2.72 versus 8.15 under OP0 for the same cells.Zero-dose cells scored 6.93 under OPP versus 7.61 under OP0; zero-dose cells comprised 55 of 450 OPP cells (12.2%).Source: C Injection Fidelity: Full Analysis · p. 16
• The fault-dose coefficient was small and slightly positive, not evidence of a dose-response: +0.06 points per fired fault, with 95% CI [−0.004, 0.12].Dose is confounded with tool-call volume, and most runs receive one permanent draw that fires from the first tool call; the authors interpret the evidence as an exposure effect.Source: C Injection Fidelity: Full Analysis · pp. 16–17

• Across all model-stack combinations, OP0 completion rates are 95 to 100% and proxy-off rates are 93 to 100%, while per-stack intervals overlap widely.The largest OPP drops are for the 120b under MCPBench and OpenAI.Source: D Supplementary Analyses of Fault Impact · p. 17
• Task degradation ranges from −19% for unit_converter task completion to −98% for game_trends grounding; live-API and deep-sequential tasks are most vulnerable.Short-chain computational tasks show the most resilience; Figure 7 pools across stacks, models, and seeds for 25 tasks and nine metrics.Source: D Supplementary Analyses of Fault Impact · pp. 17–18
• Table 5 summarizes nine pooled score-degradation metrics, with paired-drop 95% confidence intervals within ±0.4 points and corrected Wilcoxon p < 0.001 for every metric.Task fulfillment is counted only when tool appropriateness is at least 2.Source: D Supplementary Analyses of Fault Impact · p. 17, Table 5 · p. 18
• Task-completion score drops by 57%, from 8.26 to 3.53, among cells that completed under both OP0 and OPP.The primary analysis reports a 60% drop; the three-percentage-point difference indicates that quality loss among completed runs accounts for most of the effect.Source: D Supplementary Analyses of Fault Impact · p. 18
• Detailed case studies for all six failure categories appear in Appendix E, with representative reasoning traces illustrating agent responses to injected faults.Source: D Supplementary Analyses of Fault Impact · p. 18

E Failure Mode Case Studies

• Persistent access-denied errors led car_price_evaluator_000 and metropolitan_museum_000 to exhaust their turn budgets without producing scorable output.By contrast, bibliomantic_000 and metropolitan_museum_000 also illustrate catastrophic completion: they returned responses but scored 0.0 across all metrics.Source: E Failure Mode Case Studies · pp. 18–19
• Some agents completed without relying on tools: openapi_explorer_000 scored 4.4 on task completion but 0.8 on tool appropriateness, while unit_converter_000 scored 9.8 on task fulfillment but 1.2 on tool appropriateness.The unit-conversion task received no injected fault because the agent made no tool call; plausible output could therefore conceal that designated tools were never consulted.Source: E Failure Mode Case Studies · p. 18
• Partial degradation was illustrated by dex_paprika_000 (task score 0.4) and fruityvice_000 (task score 1.9), whose mixed-quality responses were poorly grounded.The case studies cover six categories and use examples from the langchain/120b condition under OPP, seed 1.Source: E Failure Mode Case Studies · pp. 18–20

F Agent Stack Orchestration Details

• The loop caps were 40 iterations for MCPBench and OpenAI and 25 LangGraph supersteps for LangChain; the timeout binds before MCPBench or LangChain reaches its cap.A small fraction of OpenAI runs (14 of 450, about 3%) reached max_turns=40 before timeout.Source: F Agent Stack Orchestration Details · p. 21
• MCPBench’s planner produced a switch-dominant profile (68% switch, 21% retry, 11% fallback), while OpenAI and LangChain were fallback-leaning (49% and 44% fallback).The stacks differ in how stopping is represented: MCPBench replans the tool list each round, while OpenAI and LangChain stop when the assistant emits no tool calls.Source: F Agent Stack Orchestration Details · p. 21, Table 6 · p. 21

G Behavioral Adaptation: Reasoning Traces and Retry Loops

• Across 1,302 fault-adjacent calls, MCPBench switched tools after 68% of failures, while LangChain and OpenAI fell back after 44% and 49%, respectively.Tool-call counts changed little between OP0 and OPP within each stack (paired Wilcoxon, p ≥0.09); the injection changed which tool agents called next, not how much they tried to use tools.Source: G Behavioral Adaptation: Reasoning… · p. 22, Figure 8 · p. 22
• In examined MCPBench/120b traces, agents hypothesized parameter or format problems after failures; precise diagnosis distinguishing permanent from transient errors was absent.This pattern is consistent with agents treating failures as potentially transient and with the observed switch- and retry-heavy responses.Source: G Behavioral Adaptation: Reasoning… · p. 22
• Of 38 runs with at least three consecutive calls to the same failing tool, 25 were in MCPBench, 7 in LangChain, and 6 in OpenAI.The longest chain comprised 12 consecutive retries over 22 tool calls and 47,761 tokens; another consumed 107,849 tokens across 52 calls.Source: G Behavioral Adaptation: Reasoning… · p. 22

H Task-Domain Sensitivity

• Chain depth was not significantly associated with per-task resilience (ρ = 0.29, p = 0.17, n = 25), and its residual association after accounting for fault dose was also nonsignificant (ρpartial = 0.35, p = 0.09).Chain depth correlated with mean fault dose (ρ = 0.47, p = 0.02), so the authors do not claim that depth causally drives vulnerability.Source: H Task-Domain Sensitivity · p. 23, Figure 9 · p. 23
• Resilience medians ranged from 0.04 for stateful financial tasks to 0.61 for multi-step sequential tasks, but the six-domain comparison was marginal (H = 9.75, p = 0.08).Pure-computation tasks had a median of 0.60; stateless retrieval sat at 0.18. With only two tasks in the pure-computation and stateful-financial classes, the authors treat the comparison as descriptive, not confirmatory.Source: Figure 9 · p. 23, H Task-Domain Sensitivity · p. 23

I Reproducibility Details

• The final 1,350 reported cells required about 58 wall-clock hours of agent execution plus about 18 hours of judge API time; the five-round campaign used about 237 wall-clock hours.The campaign reran cells that had failed under earlier, tighter timeout or round limits.Source: I Reproducibility Details · pp. 23–24

J Licenses of Used Assets

• MCP-Bench, the OpenAI Agents SDK, LangChain/LangGraph, and the gpt-oss backbones retain their stated upstream licenses; the paper’s source code is Apache 2.0 and data artifacts are CC-BY-4.0.Third-party license mappings and pinned dependency versions are documented in the artifact repository.Source: J Licenses of Used Assets · p. 24

2. Limitations

• The limitations include judge stochasticity, reporting bias in the mined fault profile, omitted fault types, and entangled exposure and error handling; task-level comparisons are also limited by three seeds and zero-dose cells.External validity is bounded by one model family and serving setup, 25 single-server tasks, and an OPP profile reflecting the open-source MCP ecosystem as of September 2025; the study is powered for a 91%-permanent incident mix rather than transient or intermittent regimes.Source: §2 · pp. 25–26

NeurIPS Paper Checklist

• The artifact provides the injector, calibrated fault profile, campaign runners, 1,350-cell results, and analysis notebook, with instructions mapping paper figures and tables to reproducing scripts.Code is released under Apache 2.0 and data artifacts under CC-BY-4.0.Source: §2 · pp. 27–28
• The paper reports effect sizes and 95% confidence intervals or significance tests for main findings, using paired designs, three seeds, and median scores across five judge evaluations per run.It uses nonparametric tests for skewed or bounded outcomes and controls multiple comparisons with Bonferroni or Benjamini-Hochberg corrections.Source: §2 · pp. 29–30

2. Limitations

• The study reports NeurIPS Code of Ethics compliance, no human subjects or personal data, and a non-adversarial proxy that reproduces naturally occurring open-source failure patterns.It also reports full disclosure of the experimental recipe, statistical tests, and validity threats.Source: §2 · p. 31

10. Broader impacts

• The broader-impacts discussion identifies reproducible pre-deployment stress testing as a benefit and flags tool-call data exposure, profile miscalibration, and adversarial repurposing as risks.It describes a null-injector default, fault-header tagging, and pre-deployment-only framing as mitigations.Source: §10 · p. 31
• The release is restricted to research and dependability evaluation, logs fault activations, and provides aggregated metadata and failure classifications rather than raw scraped threads.The proxy is scoped to MCP deployments the operator controls and cannot attack third-party servers, harvest credentials, or persist beyond a session.Source: §10 · p. 32
• LLMs serve as the evaluated agent backbones, the completion judge, and the classifier for 57,240 threads, whose outputs underpin the 18-rule fault profile.The classifier’s reliability was checked by an independent human audit, with Cohen’s κ = 0.93 for failure mode and κ = 0.80 for persistence.Source: §10 · p. 35