PaperlayerView exact revision on arXiv

Source-linked AI summary

What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead

Haseeb Mohammed Afsar

arXiv:2609.10962v1cs.SEcs.AI

TL;DR

Existing MCP studies often select or repair servers before measuring them, leaving the unrepaired registry population and benchmark-corpus repetition less well characterized. This paper samples 400 npm/stdio servers with a published seed, probes them once, and compares their advertised tools with BFCL v4 and UltraTool using one deduplicated similarity method. The random sample included 48.8% versus 66.7% for a curated frame, while real-tool near-duplication was confined within servers and BFCL showed substantial cross-task repetition.

  • Problem

    Prior MCP behavioral studies select or repair servers, while duplication within benchmark corpora and their relation to deployed tools had not been measured.

  • Method

    The paper draws 400 registry servers with a published seed, probes each once without repair or credentials, and compares tool descriptions using globally deduplicated TF-IDF cosine similarity.

  • Results

    48.8% of the random sample was included versus 66.7% of a curated frame; real-tool near-duplicates stayed within servers, while BFCL showed cross-task redundancy.

  • Takeaways & Limitations

    Curated or repaired samples cannot report how many published servers are inert, and benchmark redundancy must be separated from genuine tool diversity.

  • Takeaways & Limitations

    The behavioral tier covers only npm/stdio servers, and TF-IDF lexical similarity may miss semantic equivalence; unequal bias across corpora was not shown.

Abstract

from arXiv · show

Studies of the Model Context Protocol (MCP) server ecosystem draw their samples in ways that quietly select for servers that work: reference sets, popularity lists, hand-curated frames, or pipelines that repair a server until it starts. We report what an unrepaired probability sample actually contains. From a 24,135-server registry census we draw 400 npm/stdio servers with a published seed and probe each one over the wire. Only 48.8% complete an initialize handshake, against 66.7% for a hand-curated frame measured with the same instrument, and the dominant failure is not missing credentials (13.3%) but servers that never start at all (37.5%). Among the 195 that do run, hard conformance is total: zero fatal JSON Schema violations across 2,766 advertised tools. Optional safety annotations are the real variance, and the tool-level omission rate on a random draw is 58.8% against 41.5% on the curated frame, so curation flatters this figure too. We then compare the tool descriptions these servers advertise against two tool-use benchmark corpora using one method held constant. Real MCP tools show 2.8% near-duplication at cosine 0.70, and all of it lies within single servers: cross-author near-duplication is 0.0% at every threshold tested. BFCL v4 shows 16.7%, of which 16.4 points lie between independently presented tasks. UltraTool shows 0.3%, cleaner than real tools, so this is a property of BFCL and not of synthetic corpora as a class. Separately, 68.8% of raw BFCL rows and 85.6% of raw UltraTool rows are exact name-plus-description repeats, against 0.4% for real MCP, so any statistic computed over these releases without global deduplication measures repetition rather than tools. All figures regenerate from released scripts and a published seed.

1 Introduction

The paper measures what an unrepaired random sample of MCP servers actually contains, rather than selecting servers that already work. It also compares deployed tool descriptions with benchmark corpora using a common redundancy analysis.

  • 1 Introduction: Servers that never start were the dominant exclusion, occurring nearly three times as often as credential-gated servers.The study reports this as the failure mode that repair-based pipelines remove from view.
  • 1 Introduction: Among running servers, zero fatal schema violations occurred across 2,766 advertised tools, while safety-annotation omissions were 17.3 points higher for random than curated servers.This separates hard conformance from optional safety metadata.
  • 1 Introduction: Real deployed tools, benchmark corpora, and their within-unit versus cross-unit duplication patterns are compared using one controlled analysis.The comparison distinguishes a project repeating its own tools from benchmarks repeating independently presented tasks.
  • 1 Introduction: Every reported number is intended to regenerate from a committed script, seed, and hash-pinned sampling frame.This makes the sampling and analysis pipeline re-runnable.

2 Related work

Prior work established MCP ecosystem scale, runtime behavior, security, and source-code cloning, but used different sampling or analysis units. This paper positions its contribution as measuring unrepaired population contents and benchmark-corpus duplication.

  • 2 Related work: Prior registry work measured drift across 120 snapshots, whereas this study uses two snapshots only to size the population and construct a sampling frame.The paper explicitly makes no drift claim.
  • 2 Related work: MCPZoo repairs collected servers for dynamic analysis, while this study leaves sampled servers unrepaired to measure dead-on-arrival entries.The two approaches answer complementary questions.
  • 2 Related work: Prior security work audits internet-facing servers and reports vulnerabilities or rapid disappearance, whereas this study makes no security claim.Its behavioral sample is much smaller than MCPZoo’s.
  • 2 Related work: Repository-level cloning studies analyze source code, while this paper analyzes advertised tool interfaces and treats the difference as a validity issue.The paper tests that interface-level duplication question directly.
  • 2 Related work: Existing benchmark audits examine evaluator reliability, but the paper identifies duplication within benchmark corpora and comparison with deployed tools as an open measurement gap.This shifts attention from scoring machinery to the scored corpora.

3 Method

The method combines a metadata-only registry census, seeded probability sampling, one-shot wire probing, and globally deduplicated TF-IDF similarity analysis. It records both server outcomes and tool-level redundancy under fixed operational definitions.

  • 3 Method: The census sweeps registry metadata without executing third-party code, while the dynamic tier probes every drawn server once and records an outcome.The pipeline uses no repair, credentials, or retry.
  • 3 Method: The sampling frame contains 7,258 active npm-published, stdio-declared servers, from which n = 400 are drawn using seed 20260819.The manifest hashes the exact frame bytes to detect redraws against a changed population.
  • 3 Method: Each package is launched over stdio and tested through initialization, tool enumeration, and JSON Schema validation without invoking side-effecting tools.Every draw receives an outcome regardless of success or failure.
  • 3 Method: Redundancy is computed after global exact-key deduplication using corpus-specific TF-IDF word unigrams and bigrams with cosine similarity.The redundancy rate is the share of deduplicated tools having another tool at or above threshold t.
  • 3 Method: The corpora comprise 2,766 real MCP tools, 8,726 BFCL records, and 14,084 UltraTool records, with source files pinned by SHA-256.Exact-duplicate rates are reported separately from near-duplicate rates.

4 RQ1: what an unrepaired random draw contains

An unrepaired probability sample includes substantially fewer servers than a curated frame, with startup failure—not credential gating—the dominant exclusion. Curation raises apparent start health while lowering annotation omissions, flattering both headline measures in opposite directions.

  • Curation produced a higher start rate and a lower annotation-omission rate than the probability sample.Figure 2 places these opposite-direction effects side by side, showing why curation is difficult to correct for afterward.
  • 66.7% of curated servers started versus 48.8% of the random draw, a 17.9-point selection gap measured with the same instrument.Table 1 records the outcome of all 400 draws without silently dropping servers.
  • 37.5% of draws were servers that never started, compared with 13.3% blocked by credentials.Startup failures accounted for 150 draws, nearly three times the 53 credential-gated draws.
  • Repair-based pipelines eliminate the very non-starting servers needed to measure how much of the published population is inert.Such pipelines remain sound engineering choices for their purposes but cannot report this quantity.

5 RQ2: what the servers that do run look like

The 195 servers that ran advertised 2,766 tools with no fatal JSON Schema violations, but optional safety annotations were frequently absent. Annotation coverage appeared substantially better in the curated frame, while server-level observations support only a bounded conclusion about partial annotation.

  • Zero of 2,766 tools had a fatal JSON Schema violation, and zero of 195 servers had any.The result covers missing schemas, invalid types, malformed properties, and malformed required fields, at roughly fourteen times the scale of an earlier curated result.
  • 58.8% of tools carried no optional safety annotations, versus 41.5% in the curated frame.The random draw therefore had a 17.3-point higher omission rate, so the curated figure should not be treated as an ecosystem rate.
  • All 194 servers with at least one tool were all-or-nothing for annotations, but the sample supports only a 1.53% one-sided 95% upper bound on partial-server prevalence.An earlier unreleased run observed four partial servers among 214, so the present zero count does not establish their absolute absence.
  • Four protocol versions appeared across the 195 servers, including one version newer than the client baseline.The deployed population therefore varied in both directions relative to the advertised baseline.

6 RQ3: benchmark corpora against real deployed tools

After global deduplication, BFCL’s redundancy spans independently presented tasks, whereas real MCP’s near-duplicates remain confined within individual servers. UltraTool is cleaner than real deployed tools, showing that the benchmark pattern is corpus-specific.

  • 6.1 Raw releases are mostly repetition: Global deduplication is required because raw benchmark files repeat exact name-plus-description pairs, causing unreduced statistics to measure task repetition rather than tool counts.The authors note that the exact-duplicate percentage depends on which files are included and should be read alongside post-deduplication rates.
  • 6.2 Near-duplication, and where it lives: 16.4 of BFCL’s 16.7 points of near-duplication lie between independently presented tasks, while real MCP has 0.0% cross-author near-duplication at every threshold.Real MCP’s near-duplicates arise within single servers, such as list_x, get_x, and create_x families.
  • 6.2 Near-duplication, and where it lives: UltraTool has 0.3% near-duplication versus 2.8% for real deployed tools, so the evidence does not support a claim about synthetic tool corpora as a class.

7 Ecosystem context

The registry expanded between two complete snapshots, while the npm/stdio slice grew in absolute terms but declined as a share of the ecosystem. This limits the coverage of local behavioral instruments.

  • 7 Ecosystem context: The registry grew from 16,548 to 24,135 unique servers between 2026-07-14 and 2026-08-22, about 195 net new servers per day.Both census sweeps completed.
  • 7 Ecosystem context: Remote-only servers grew 70.1% versus 26.3% for package-only servers between the two measured endpoints, but two snapshots do not establish a trend.
  • 7 Ecosystem context: The npm/stdio slice increased from 5,804 to 7,414 servers but fell from 35.1% to 30.7% of the population.Because local behavioral probing reaches this slice, the instrument covers a shrinking minority of the ecosystem.

8 Threats to validity

The study’s behavioral and similarity results are bounded by its description-based, single-probe design and by incomplete ecosystem coverage. Several checks address selection and comparison concerns without eliminating these scope limits.

  • 8 Threats to validity: The handshake filter shows no evidence of selecting against clones, but description rewrites could conceal implementation cloning from this method.The check compared started and non-started groups, author-family concentration, and the most prolific included author’s cross-server similarity.
  • 8 Threats to validity: The cross-unit comparison is not power-matched: BFCL averages 1.8 tools per task row versus 14.2 per real-MCP server, partly preserving BFCL’s cross-task rate.This imbalance does not explain real MCP’s collapse from 2.8% to 0.0%.
  • 8 Threats to validity: The behavioral tier covers only npm/stdio servers, 30.7% of the population, and a single probe counts transient failures as exclusions.Thus 48.8% is a lower bound on the fraction that could ever start, while remote servers, PyPI, and OCI packages remain out of frame.
  • 8 Threats to validity: TF-IDF cosine measures lexical overlap rather than semantic equivalence, so tools expressing the same function in different words count as distinct.The authors state that this biases all three rates downward but have not shown that the bias is unequal.
  • 8 Threats to validity: Registry metadata is self-declared, and the two census snapshots support a measured endpoint difference rather than a trend.

9 Data and code availability

The paper releases its probing, harvesting, sampling, aggregation, similarity, and threat-testing artifacts. A published seed and per-server outcomes allow every figure and aggregate to be independently regenerated and recounted.

  • 9 Data and code availability: The open-source release includes the probe, harvester, seeded draw script, resumable runner, aggregation, redundancy measurement, and threat-test suite.The tool release and dataset are archived on Zenodo under concept DOI 10.5281/zenodo.21347997.
  • 9 Data and code availability: Every figure regenerates from a committed script, the published seed 20260819, and the draw-manifest frame hash.The released per-server outcomes for all 400 draws also permit independent recounting of the aggregates.
Loading 2609.10962v1…