Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse

1 Introduction

• Existing agent evaluations emphasize task outcomes or valid tool-use trajectories, which do not establish whether tool information is complete, qualified, represented, or interpreted correctly.In biology, provenance, evidence strength, experimental status, organism and disease context, and completeness can affect a result’s meaning.Source: §1 · p. 2
• A silent failure occurs when a tool call appears successful, its returned information is insufficient or transformed, and the limitation is not adequately disclosed downstream.Such limitations can be amplified when an agent treats partial output as complete evidence.Source: §1 · p. 2
• The central question is whether successful scientific tool interactions preserve the information, qualifiers, context, and meaning needed for the intended conclusion.The research questions also ask how prevalent failures are, where they occur, which properties are vulnerable, and whether agents preserve, disclose, or amplify limitations.Source: §1 · p. 2
• The seven-locus taxonomy distinguishes tool limitations, API gaps, API partial gaps, wrapper gaps, wrapper partial gaps, agent usability gaps, and agent interpretation gaps.The loci identify where failures originate in the interaction chain.Source: §1 · pp. 2–3

2 Experimental environment and methodology

• The study selected 15 tools with 104 associated ToolUniverse tools for detailed analysis from a random sample of 63 main tools containing over 285 associated tools.ToolUniverse was set up using official MCP guidance with the AntiGravity CLI agent using Gemini 3.5 flash lite.Source: §2 · p. 3
• Candidate discovery compared tool documentation, web interfaces, APIs, manuals, release notes, and reported GitHub issues for mismatches in functionality, fields, parameters, retrieval, semantics, provenance, and temporal context.The audit also considered whether requested information, conditions, attribution, retrieval boundaries, and meaning were preserved.Source: §2 · p. 3
• Concrete test cases used specified ToolUniverse names and scientific inputs, targeted an underlying resource rather than a prescribed wrapper, and isolated the candidate failure from independent diagnosis.Queries were iteratively refined through manual verification before automation.Source: §2 · pp. 3–4
• Each candidate was tested repeatedly, with additional executions varying concrete inputs where possible to assess whether an observed behavior generalized beyond one case.Initial failures prompted four additional queries; false-positive or unverified cases prompted two.Source: §2 · p. 4
• Cursor classified execution traces as failures, false positives, or unverified cases, while two authors manually reviewed 198 test cases across 15 randomly selected main tools.Manual review checked candidate facts against web interfaces and assessed whether the evaluator’s judgment was valid; failures were also classified across ten analysis dimensions.Source: §2 · p. 4

3 Results

• API partial gaps (L3) and wrapper partial gaps (L5) were the dominant failure loci for most tools.These loci accounted for the largest share of human-verified failures in BiGG Models (100%), BV-BRC (100%), CIViC (100%), and Reactome (88%).Source: Figure 3 · p. 5, §3 · p. 5
• 7 of 15 tools had confirmed failures in more than 50% of test cases, including ExpressionAtlas (75%), BV-BRC (67%), and AMPSphere (67%).The other listed tools were CTIS (53%), ChIP-Atlas (57%), KEGG (53%), and Reactome (53%).Source: Figure 3 · p. 5, §3 · p. 5
• Completeness led the analysis dimensions with 28 occurrences among 134 issue occurrences from 91 evaluated cases; Ranking and Relevance had 21 and Result Appropriateness had 20.Individual cases could involve multiple dimensions.Source: Figure 4 · p. 6, §3 · p. 6
• Missing data or fields was the most frequent issue type (38 occurrences), followed by search, filtering, and ranking criteria (20 instances).Individual test cases could exhibit more than one issue.Source: Figure 4 · p. 6, §3 · p. 6
• Examples show upstream limitations becoming downstream claims, including API capping becoming a completeness claim and missing filtering becoming inappropriate inclusion.The agent generally disclosed loud failures but often failed to disclose silent failures such as possible truncation from wrapper limits.Source: §3 · p. 6

L1: Tool Limitation

• AMPSphere’s “Blanks” and “Others” lack further classification, while ChIP Atlas similar-match search can return irrelevant results when it conflates a molecule with its post-translational form.The examples also include Progenome being misclassified as a habitat rather than a genomic reference database.Source: §4 · p. 7

L2: API Gap

• ChIP Atlas differential peak analysis and KEGG Mapper functions available through web interfaces lack corresponding API endpoints, blocking their use through the agent.The missing KEGG Mapper operations include Reconstruct, Color, and Join.Source: §4 · p. 7

L3: API Partial Gap

• BiGG’s API returns a null organism field for model iRC1080, and AMPSphere’s API omits antimicrobial and hemolytic activity probabilities shown in its Web UI.The examples concern information available elsewhere in the tools but absent from API responses.Source: §4 · p. 7

L4: Wrapper Gap

• KEGG’s documented BlastKOALA/GhostKOALA capability is absent from its REST API, while AMPSphere’s wrapper omits global statistics available from the raw API.The missing statistics include family counts and metagenome totals.Source: §4 · p. 7

L5: Wrapper Partial Gap

• The BVBRC wrapper filters out external identifiers, and ClinVar and CIViC wrappers omit fields needed to distinguish contributing records or expose available variant information.The ClinVar field identifies submissions contributing to aggregate classification; the CIViC API schema includes MyVariantInfo fields.Source: §4 · p. 7

L6: Agent Usability Gap

• No L6 issue was found in manual review, but some responses approached the agent’s context-window size or supplied image data in formats requiring further processing.The authors note that multi-tool, multi-step test cases may expose usability failures more clearly.Source: §4 · p. 7

L7: Agent Interpretation Gap

• The agent misread ChIP Atlas’s logarithmic significance score as generic confidence and misunderstood KEGG’s rule that each top-level parenthetical step needs at least one satisfied member.The ChIP Atlas score is documented as -10 * log10(Q-value); in KEGG modules, commas represent alternatives and spaces or plus signs represent connections.Source: §4 · pp. 7–8

Why silent failures occur?

• The study characterizes ecosystem-level contributors—rapid component evolution, distributed responsibility, changing resources, and partial API or wrapper functionality—as plausible factors, not causal explanations for every case.The authors say the study was designed to identify and characterize failures, rather than establish their engineering or organizational root causes.Source: §5 · p. 8

What does it mean?

• The interaction pathway, rather than any one API, wrapper, or agent, is the unit of analysis because information provenance and transformations shape how outputs should be interpreted.A successful tool call alone does not establish that the task was completed correctly or that the returned result was used faithfully.Source: §5 · p. 8

Why does this matter?

• Truncation, unavailable filtering, missing metadata, and ambiguous semantics can propagate through a workflow, allowing partial results or omitted qualifiers to support stronger downstream claims than the evidence warrants.In biological workflows, evidence type, organism specificity, and search scope can affect interpretation.Source: §5 · p. 8
• Silent failures can burden users with verification, offset efficiency through rework, and yield fluent answers without establishing factual correctness or adequate evidence.Composed workflows can carry limitations into later retrieval, analysis, or interpretation unless provenance is retained.Source: §5 · pp. 8–9

What can be done?

• The authors propose contextual reliability measures, automated preservation tests, runtime disclosures, limitation propagation, repeated deployment testing, machine-readable tool disclosures, and ecosystem monitoring.The proposed tests cover matters such as API availability, parameter mapping, field preservation, filtering, pagination, formats, and error handling.Source: §5 · p. 9

Our proposition of contextual reliability

• Contextual reliability measures whether an interaction preserves and communicates the information, qualifiers, provenance, scope, and meaning needed to support its intended scientific conclusion.A successful call or valid data return alone does not establish contextual reliability.Source: §5 · p. 9

6 Limitations and conclusion

• The audit covers a ToolUniverse-specific environment, deeply analyzes 15 tools and 104 wrappers, and uses LLM-assisted candidate generation with manual validation of only a subset.Tool and interface changes may also alter individual observations, and the authors describe their approach as non-exhaustive.Source: §6 · p. 9
• The authors characterize silent failures across 15 scientific tools and 198 candidate test cases, using a taxonomy that spans tool limitations, API and wrapper gaps, agent usability, and agent interpretation.ToolUniverse is the experimental environment, not the object of the study.Source: §6 · p. 9
• The study proposes contextual reliability to assess whether information, qualifiers, provenance, scope, and meaning remain sufficient for the intended scientific conclusion.The authors present the results as observed risks, not an exhaustive or representative estimate of all agent–tool failures.Source: §6 · p. 9

A Failure Measurement

• The rubric includes correctness, appropriateness, consistency, reliability, and attribution as failure-measurement dimensions.Checks include factual and semantic accuracy, tool and parameter selection, repeatability, error handling, and provenance.Source: Table 1 · p. 12, A Failure Measurement · p. 13
• The remaining dimensions cover interpretation, relevance, context and conditions, and purpose alignment.These dimensions address ranking effects, qualifier preservation, query and scope alignment, and fitness for downstream use.Source: Table 1 · p. 12, A Failure Measurement · p. 13
• Verification checks include authoritative-source comparisons, appropriate tool and endpoint selection, repeat-query consistency, execution failures, provenance traceability, and exposed ranking logic.The rubric asks whether truncation or partial responses are surfaced and whether source and evidence identifiers survive the chain.Source: A Failure Measurement · p. 13
• The rubric also checks whether qualifiers and symbols are interpreted correctly, answers match the request, and outputs suit the intended use.It asks whether qualified claims become absolute, irrelevant information replaces requested information, or technically valid answers could mislead downstream conclusions.Source: A Failure Measurement · p. 13

B Dimensions, Failure Loci, and Representative Instances

• The examples span mismatches in retrieval and ranking, interpretation, attribution, and reliability across the listed failure loci.Examples include accession-naming drift and unclear tool-version transparency, missing attribution fields, and inconsistent result caps or sorting.Source: Table 2 · p. 13, B Dimensions, Failure Loci, and… · p. 14
• The KEGG agent misapplied module-completeness logic, while BV-BRC’s wrapper omitted quality metrics returned by its API.The KEGG definition requires at least one present K-number in every parenthetical group; the wrapper omission hides checkm completeness, contamination, and genome-quality fields.Source: B Dimensions, Failure Loci, and… · p. 14
• Examples of interpretation failures include misreading a MACS2 transformed score as linear and receiving expression values without TPM/FPKM units.The cited cases also include ambiguity about whether exact-match searches return homology-based hits.Source: B Dimensions, Failure Loci, and… · p. 14
• Attribution examples include empty PMID fields, a habitat category that represents a genome database source, and wrapper omissions of accession fields.The reliability examples also include accession conventions that drift over time and a missing InterProScan version that affects reproducibility and meaning.Source: B Dimensions, Failure Loci, and… · p. 14

C.1 AMPSphere

• AMPSphere’s “Blank” and “others” microbial-source categories lack definitions specific enough for environmental interpretation.The categories do not provide sufficiently specific semantic definitions.Source: §C.1 · p. 15
• Structure and family raw data available in AMPSphere’s web UI are missing from the API, leaving the agent without potentially relevant interpretive factors.Source: §C.1 · p. 15
• The agent interpreted sample quality without considering all available quality scores.Source: §C.1 · p. 15

C.2 BV-BRC

• BV-BRC’s wrapper omits CheckM scores and genome-quality status, while its API baseline lacks assembly information present in the web UI.The wrapper also omits reference/representative-genome status, so the agent may select a reference without source evidence for that designation.Source: §C.2 · p. 15

C.3 ClinGen

• ClinGen’s documented validity endpoints do not expose the web interface’s drug-name/RXNORM search, and its wrapper does not accept genomic coordinate strings.Source: §C.3 · p. 15

C.4 CIViC

• CIViC’s API can flatten distinctions between individual Evidence items and synthesized Assertions, and its Boolean and ontology-expanded web search is not equivalently exposed.A parent-disease search may therefore fail to retrieve evidence associated with specific subtypes.Source: §C.4 · p. 15

C.5 ChIP-Atlas

• ChIP-Atlas normalizes phospho-SMAD3 to SMAD3, lacks a differential-analysis endpoint, and incompletely specifies search and ranking semantics.The agent also interpreted a MACS2 score of 500 as a raw count rather than the documented −10 log10(q) transformation.Source: §C.5 · p. 15

C.6 CryoET Data Portal

• CryoET publication metadata may be returned as null, and the wrapper omits the API’s child-dataset field for depositions.The omission prevents the agent from discovering the hierarchical relationship between a deposition and its linked datasets.Source: §C.6 · pp. 15–16

C.7 EMDB

• EMDB search results are capped at 1,000 rows, and its REST API supports basic keyword search rather than the web interface’s documented advanced field filters.Source: §C.7 · p. 16

C.8 BiGG Models

• BiGG search and detail responses can differ in metadata, its API offers less advanced search than the web interface, and wrappers omit gene-sequence and strand fields.The wrapper provides high-level model metadata but no functionality for accessing gene-level details returned by the underlying API endpoint.Source: §C.8 · p. 16

C.9 ClinicalTrials.gov

• ClinicalTrials.gov returns success with empty outcomes but no “No Results Posted” flag, and an overly long query for “FDAAA 801 Violations” returns no results.The study identifies “FDAAA 801” alone as the correct query term.Source: §C.9 · p. 16

C.10 CTIS

• In CTIS, the agent misinterpreted result completeness because of pagination and treated API search as equivalent to web-interface Boolean filtering.Source: §C.10 · p. 16

C.11 ClinVar

• ClinVar’s wrapper omits submission-level observation metadata and the “C”/“N” badges that distinguish submissions contributing to aggregate classifications.Source: §C.11 · p. 16

C.12 Expression Atlas

• Expression Atlas’s API lacks a programmatic endpoint for gene-level expression values and may omit project affiliations, citation fields, and TPM/FPKM units.Experiment accessions can also differ between the web interface and API, and the tool may not resolve the web-interface accession or provide alias mapping.Source: §C.12 · p. 17

C.13 InterPro

• InterPro’s API lacks the web interface’s raw-sequence and domain-architecture searches, and exposes less taxonomic lineage information.Source: §C.13 · p. 17

• The Content Service lacks a per-instance referrers endpoint, so agents must reconstruct parent relationships through child eventOf links, potentially making N+1 queries and limiting complete hierarchy traversal.Source: §C.15 · p. 17
• Reactome services use different species identifiers: Content Service operations may use NCBI taxonomy IDs such as 9606, while Analysis Service comparisons require Reactome dbId values such as 48892.Source: §C.15 · p. 17
• Reactome search clusters results by default, but the ToolUniverse interface exposes no cluster parameter, and the API limits q to 200 characters.Source: §C.15 · pp. 17–18
• Reactome’s Reaction records include binding and translocation as well as other biological state changes, so they are not necessarily conventional biochemical conversions.Source: §C.15 · p. 18