PaperlayerView exact revision on arXiv

Source-linked AI summary

MCP Error Messages Written for Developers Hurt the Most Capable Agents Most

Xiaonan Xu, Wenjing Wu

arXiv:2609.35381v2cs.SEcs.AI

TL;DR

MCP error messages often assume their reader can act outside a tool call, although many agents can only call the connected server’s tools. The paper compares five models’ recovery across error-message conditions, including original and rewritten next steps that name server-tool repairs. Naming a server tool raised recovery on expired credentials from 45% to 84%, while removing the step raised it to 82%; naming the repeated call raised rate-limit recovery from 6% to 88%.

  • Problem

    MCP error messages often assume their reader can act outside a tool call, although many agents can only call the connected server’s tools.

    Source: §1 · p. 2

  • Method

    The paper compares five models’ recovery across error-message conditions, including original and rewritten next steps that name server-tool repairs.

    Source: §1 · p. 3, §4.2 · p. 5

  • Results

    Naming a server tool raised recovery on expired credentials from 45% to 84%, while removing the step raised it to 82%; naming the repeated call raised rate-limit recovery from 6% to 88%.

    Source: §1 · p. 3, §5.1 · p. 9, §6 · p. 12

  • Takeaways & Limitations

    A server-tool instruction is usable by callers with different capabilities, while removing a step is useful for credential errors when the cause already points to the repair.

    Source: §6 · p. 12

    The evaluation uses BFCL multi-turn scenarios spanning seven failure types, and the server survey covers eligible open-source GitHub servers in September 2026.

    Source: §7 · p. 12

Abstract

from arXiv · show

Many Model Context Protocol (MCP) servers wrap web APIs built for human developers, and their error messages tell the reader to run a command, edit a configuration, open a web page or wait. Many agents that read them can only call the server's tools. In 150 widely used MCP servers, 949 of 3,001 error messages tell the caller what to do next, and half of these steps depend on something the server cannot see about the caller. On credential errors, 62 of 67 steps ask for a terminal command, a configuration change or a web page; on rate limits, 20 of 30 say to wait and retry without naming the call to repeat. We tested five OpenAI models that act only through the tools of Berkeley Function Calling Leaderboard tasks, and the agents did what the step said. On expired credentials, a terminal command in the step left 45% of tasks recovered, and the loss it caused grew from 18 points for GPT-5.5 to 69 for GPT-6 Astra. On a rate limit, GitHub's "Wait before retrying." left 6%. We tested two remedies. For MCP developers, naming a server tool in the step raised recovery on expired credentials to 84%, with the login tool in place of the command, and on a rate limit to 88%, with the call to repeat in place of the bare wait. For agent developers, deleting the step with a one-sentence prompt before the model reads it raised recovery on expired credentials to 82%.

1. Introduction

MCP error messages often address a human developer who can act outside a tool call, while many agents can only call the connected server’s tools. The paper surveys these messages, tests how tool-only agents respond, and evaluates remedies for both server and agent developers.

1. Introduction

  • Many MCP servers wrap web APIs whose error messages assume a reader can run commands, edit configuration, use a web page, or wait.Of 116 official servers, 88.6% are backed fully or partly by REST APIs, and 92% implement their tools as bare API wrappers [18].Source: §1 · p. 2
  • A tool-only agent cannot carry out many developer-directed steps, and the server cannot tell which capabilities its caller has.A call to one of the server’s own tools is the kind of step every caller can carry out.Source: §1 · p. 2
  • The paper measures error-message steps across 150 servers, tests agent responses in replayed BFCL tasks, and evaluates remedies for server and agent developers.The server-side remedy rewrites a step as a server-tool call; the agent-side remedy removes the step before the model reads it.Source: §1 · pp. 2–3
  • 45% of tasks recovered with an expired-credential terminal-command step; naming the login tool raised recovery to 84%, and removing the step raised it to 82%.On rate limits, naming the call to repeat raised recovery from 6% to 88%.Source: §1 · p. 3

2. Related Work

Prior work studies developer-facing errors, tool feedback, and agents’ responses to instructions in tool output. This paper examines whether the reader of an MCP error message can act on its suggested step.

2. Related Work

  • Research on developer-facing error messages includes usability studies and findings that compiler messages are often unhelpful.HTTP APIs also return machine-readable errors separating the problem type from human-readable detail.Source: §2 · p. 3
  • For coding agents, removing detail from type-error messages lowers repair rate; tool feedback helps models revise outputs, while training models to diagnose failed calls improves recovery [25] [8] [11] [10].Source: §2 · p. 3
  • Agents carry out instructions in tool output, and prior work reports that the most capable models reliably follow instructions in MCP tool descriptions [1] [12] [13].This paper studies whether the message’s reader can act on its step.Source: §2 · p. 3
  • The authors report no known study of MCP servers’ returned error text or agents’ responses to it.Prior MCP ecosystem studies covered faults, tool-description defects, and REST API wrapping [22] [15] [18].Source: §2 · p. 4

3. Error Text in Deployed MCP Servers

The survey examines 3,001 error messages from 150 widely used MCP servers, recording whether each suggests a next step and whether that step depends on caller conditions the server cannot observe.

3. Error Text in Deployed MCP Servers

  • Of 3,001 messages, 949 tell the caller what to do next, and half of those steps depend on the caller.The servers were the 150 most-starred eligible GitHub projects, and OpenAI Codex agents labeled messages from source code using a released codebook.Source: §3 · p. 4
  • 99 of 128 steps on credential, permission, and rate-limit errors depend on the caller; 93 ask for a configuration change, web-page action, terminal command, or waiting.Source: §3 · p. 4
  • 62 of 67 credential-error steps ask for a configuration change, web page, or terminal command, although 12 such cases offer a server tool that could make the repair.Source: §3 · p. 4
  • 20 of 30 rate-limit steps say to wait and retry, and none names which call to repeat.The paper tests these credential and rate-limit steps in its experiment.Source: §3 · p. 4

4. Experiment

The experiment replays BFCL tasks from saved states after a tool call fails, then compares recovery under different error texts. Its scenarios, text conditions, model runs, and analysis define the scope and measurement of the comparison.

4.1. Scenarios

  • 168 BFCL scenarios cover seven failure types, with 24 scenarios per type and recovery judged against the task’s expected service state.The types include malformed calls, expired credentials, missing resources or permissions, and rate limits.Source: §4.1 · p. 5, Table 2 · p. 6

4.2. Error text

  • Each scenario has six texts across four conditions, including cause-only notices, correct next steps, and executable or unavailable incorrect steps.The original and rewritten steps reproduce surveyed suggestions and rewrite the repair as a server-tool call; the tested originals include a terminal command and “Wait before retrying.”Source: §4.2 · p. 5, Table 3 · p. 7

4.3. Models and procedure

  • Five OpenAI models from three generations were evaluated, with three samples per scenario, text, and model; the agents could call BFCL tools but had no terminal, clock, or browser.The expired-credential filter test added 1,440 trials after filtering 96 texts with GPT-6 Luna.Source: §4.3 · pp. 5–7

4.4. Analysis

  • Recovery means passing the BFCL state check, and comparisons use scenarios as the unit with 95% intervals from 10,000 bootstrap resamples within failure type.The same resampled scenarios were used for every text of a model.Source: §4.3 · p. 7, §4.4 · p. 7

5. Results

Across expired credentials and rate limits, steps that ask for an unavailable action or only say to wait leave many trials unrepaired; naming an available repair improves recovery. A prompt that removes such steps also restores expired-credential recovery, while failures within the agent’s own call show little change across texts.

5.1. Expired credentials: a step that asks for a terminal command

  • 45% recovery with a terminal-command step, down from 82% with the cause alone; naming the login tool gave about the same recovery as the cause alone.Table 4 compares the texts across 24 expired-credential scenarios.Source: Table 4 · p. 8, §5.1 · p. 8
  • The command-related recovery loss grew from 18 points for GPT-5.5 to 69 for GPT-6 Astra.GPT-5.6 Sol lost 35 points, GPT-6 Sol 39, and GPT-6 Luna 26; GPT-6 Astra exceeded GPT-5.5 by 51 points (interval 29 to 72).Source: §5.1 · p. 9

5.2. Rate limit: a step that says to wait

  • Naming the call to repeat raised rate-limit recovery by 82 points (interval 75 to 88), to between 78 and 99% per model.Under “Wait before retrying.”, 89–99% of trials ended without repair; Table 5 compares both phrasings across 24 scenarios.Source: §5.2 · p. 9, Table 5 · p. 9

5.3. Removing the steps with a prompt

  • Filtering the terminal-command step raised expired-credential recovery by 37.5 points on average, matching recovery under the cause alone.Where a removed step had been correct, recovery changed by −1.7 to −0.8 points, with every interval including zero; the prompt cost $0.09 to run on 949 messages with a step.Source: §5.3 · p. 10

5.4. The other failure types

  • For failures in the agent’s own call, recovery under every text stayed within four points of the generic notice; naming the login tool raised missing-permission recovery to 53%, versus 28% for re-authorization wording.Table 6 reports recovery by failure type, averaged over five models.Source: §5.4 · p. 10, Table 6 · p. 10

6. Discussion

MCP error steps can make agents abandon repairs their own tools could perform, especially when the step asks for an unavailable action. Tool-specific instructions and selective removal offer remedies, while newer models’ stronger attention to written instructions may increase the cost of developer-oriented steps.

Why text written for developers fails these agents

  • Agents followed tool-naming steps but mostly stopped when instructed to do something unavailable, despite having tools that could make the repair.The error text mattered when the failure was outside the agent’s call; malformed calls showed agents what to change regardless of the message.Source: §6 · p. 11

Why the newest model stops most often

  • GPT-6 Astra often abandoned expired-credential repairs after a sentence asking for a terminal command, despite recovering in most trials given only the cause.The authors call this literal compliance: an agent unable to carry out a suggested step gives up a repair its own tools could make.Source: §6 · p. 11
  • The same server text may cost newer agents more recovery because error steps are written by tool authors whom the agent developer does not control.The authors relate this concern to reports that newer models weigh instructions more heavily.Source: §6 · p. 12

What an MCP developer can do

  • Naming a server tool raised recovery on all three failures it could repair; on rate limits, naming the call to repeat is the essential instruction.Agents repeated the failed call even when the step named the wrong tool.Source: §6 · p. 12

What an agent developer can do

  • On expired credentials, deleting steps raised recovery by 37.5 points, while removing a correct step for missing permissions or rate limits would remove the repair.The approach suits credential errors because the cause already points to the repair.Source: §6 · p. 12

7. Threats to Validity

The evaluation covers seven failure types in BFCL multi-turn domains, with recovery judged for the turn containing the failure.

7. Threats to Validity

  • Recovery is judged by the BFCL state check for the turn in which the failure occurs.The check does not describe recovery beyond that turn.Source: §7 · p. 12
  • The scenarios cover seven failure types across BFCL multi-turn domains.Source: §7 · p. 12
  • The survey describes widely used open-source MCP servers on GitHub in September 2026.Source: §7 · p. 12

8. Conclusion

The paper finds that developer-oriented MCP error steps can leave tool-limited agents without a usable repair. It reports improved recovery from tool-specific steps and, for expired credentials, removing the step before the model reads it.

8. Conclusion

  • In 150 widely used MCP servers, next steps often ask for actions beyond MCP tools, and agents mostly stopped when given such steps.GPT-6 Astra stopped in almost every expired-credential trial.Source: §8 · p. 13
  • On expired credentials, recovery was 45% with a terminal command, 84% with the login tool named, and 82% after the agent developer deleted the step.Source: §8 · p. 13
  • On a rate limit, naming the call to repeat in place of a bare wait raised recovery from 6% to 88%.Source: §8 · p. 13

CRediT authorship contribution statement

The contribution statement assigns conceptualization, methodology, investigation, and original drafting to Xiaonan Xu, and software, validation, formal analysis, and review and editing to Wenjing Wu.

CRediT authorship contribution statement

Related papers

Loading 2609.35381v2…