SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Tools

1 Introduction

• An agent cannot distinguish an empty match from an misunderstood query when both produce parseable HTTP 200 responses without an error signal.The paper locates the problem in interface contracts that leave accepted values in prose, so syntactically valid requests can pass automated checks and return empty results.Source: §1 · p. 1

1.1 A motivating observation

• A reasonable but undocumented synonym can return zero results, while a malformed range can be silently discarded and return a plausible, overly broad result.The latter is harder to notice because its response looks well-formed despite violating the requested constraint.Source: §1.1 · pp. 1–2

1.2 Approach and contributions

• The paper frames silent failure as predictable from published artifacts, and introduces silent drop and semantic downgrade as failure modes missing from prior noise models.A silent drop returns a plausible superset after a filter is discarded; a semantic downgrade weakens the request to what the documented vocabulary expresses.Source: §1.2 · p. 2
• The study combines audits of public OpenAPI documents and an aggregation layer with executed perturbations, model tests, downstream-harm measurements, and a schema intervention.Monid supplies common-format schemas and per-call identifiers; the paper also releases its code, schemas, perturbations, transcripts, and judge labels.Source: §1.2 · p. 2

2 Related Work

• Existing agent-tool benchmarks differ in environment realism but score answer correctness, not whether the environment answered the requested question.Locally reimplemented API-Bank endpoints cannot exhibit vendor-specific parsing behavior.Source: §2 · p. 2
• Simulated-tool benchmarks can report robustness without testing silent constraint discards, because simulated responses follow endpoint documentation.StableToolBench and MirrorAPI are described as careful engineering, but their simulations do not reproduce undocumented silent discards.Source: §2 · p. 2
• Prior noise models omit silent drop: a plausible, well-formed response that violates the caller’s constraint.AgentNoiseBench and related work inject tool-side noise, but the paper argues the most dangerous observed mode is absent from the field’s model.Source: §2 · p. 2
• This work derives API probes mechanically from schemas and models plausible language-model inputs, unlike earlier hand-identified metamorphic tests.The paper also shifts the target from API-owner defect reports to environment properties relevant to agent builders.Source: §2 · pp. 2–3
• Prior studies examine specification quality or valid-input execution, whereas this work dynamically analyzes request-response behavior and its effects on consumers.The paper also notes that lenient handling of unknown or invalid parameters is discussed by practitioners but lacks a prevalence study or measurement of its effect on automated consumers.Source: §2 · p. 3

3 Methodology

• Three instruments run in order: a static audit, live differential probes, and an agent loop executing model-generated calls.Results for all three are reported in Section 4.Source: §3 · p. 3

3.1 Auditing constraint form

• The audit classifies scalar leaves as machine-checkable or documentation-constrained using schema keywords and description patterns, without issuing API calls.For array-valued parameters, descriptions must be carried down to items or vocabularies can be missed.Source: §3.1 · p. 3

3.2 Instruments

• Aggregation layers provide uniform access to independent vendors and models, while Monid supplies published endpoint schemas and structured execution records.The design also depends on freely retrievable schemas, per-call run identifiers, a shared contract, and a common validator; the endpoint and model samples are consequently gateway-dependent.Source: §3.2 · p. 3

3.3 The differential probe

• The differential probe compares a base call, a valid-value call, and a perturbed call, using result counts and row-identity fingerprints to classify rejection or silent behavior.A perturbation is a silent drop when its signature matches the no-filter base; perturbations are schema-derived across ten families designed to model plausible model outputs.Source: §3.3 · pp. 3–4
• Probe controls exclude invalid synthesized values, empty base results, and parameters inert on the query, while repeated calls help distinguish throttling and endpoint noise.Four-point failures are repeated because genuine rejection is deterministic whereas throttling is not; a per-endpoint tolerance accounts for repeat-call variation.Source: §3.3 · p. 4

3.4 Agent experiments

• Agent experiments recover lower-bound vocabularies, then compare ordinary-English tasks under the published schema and an otherwise identical schema with the vocabulary promoted to an enum.Twelve models across eight families are tested on live endpoints, and emitted calls are classified with the differential probe.Source: §3.4 · p. 4
• The agent study restricts tools to three task-relevant parameters to avoid unguessable-field failures, normalizes parameter-name drift, and evaluates final answers after up to four executed turns.The full loop feeds real responses back to the model and uses a separate judge to classify its final answer.Source: §3.4 · pp. 4–5

4.1 How common is the gap?

• Across 721,320 public parameters, 7.5% declared an enumeration and 15.2% any machine-checkable constraint; 29.4% had no description, and 40.1% of documents stated a prose constraint absent from the schema.The document-level gap was higher than the 20.9% measured on the aggregation layer.Source: §4.1 · p. 5
• The per-leaf prose-vocabulary rate did not replicate, and the authors warn the failure may be more common in hand-authored agent interfaces than in generated public specifications.They state this as a scope limit because generated specifications dominate the public corpora.Source: §4.1 · pp. 5–6

4.2 Does constraint form predict honesty?

• 44 of 61 prose-only perturbations failed silently, versus 0 of 111 machine-checkable perturbations (Fisher’s exact p = 2 × 10−13).The scored set comprised 219 perturbations over 71 parameters across 27 vendors.Source: §4.2 · p. 6
• All 113 gateway rejections named the offending parameter and legal values, while all 14 vendor rejections returned only “Invalid input: HTTP 400.”Of 111 honest errors on machine-checkable parameters, 110 came from Monid’s schema validator; those requests never reached the vendor.Source: §4.2 · p. 6

4.3 Do agents walk into it?

• 88 of 88 attempts failed when descriptions gave only an example, versus 9 of 178 for fully documented vocabularies; promoting the example-only vocabulary to an enum reduced failures to 0 of 89.For fully listed vocabularies, models emitted valid values in 23 of 23, 22 of 23, and 20 of 22 attempts across three tasks.Source: §4.3 · p. 6
• Under the lifted enum, 3 of 395 calls still emitted an out-of-enum value, showing that schemas both shape generation and let validators reject values the model emits.Only the validation role remains when a model ignores the enum.Source: §4.3 · p. 7

4.4 What does the user get told?

• Of 94 zero-row agent loops, only 11 retried with another value, and none recovered; across silent failures, detection was 12%, repair 0%, false negatives 41%, and fabricated figures 12%.The authors caution that a majority of responses hedge, so the data do not support the broader claim that agents blindly report false negatives.Source: §4.4 · p. 7

4.5 Where the effect stops

• With a closed-list vocabulary, all 23 models asked for certified pre-owned cars emitted used, returning 21 plausible wrong rows; adding the enum repaired results: 23 of 179 without it, 0 of 176 with it.Source: §4.5 · p. 7

5.1 Limitations and future directions

• The execution sample covers Monid-reachable, read-only endpoints priced at or below $0.02 per call, not a random sample of production APIs.Baseline synthesis is also more successful for parameters with enumerations or examples, which correlates with the independent variable.Source: §5.1 · p. 7
• The inert exclusion can hide genuine silent drops, so the measured silent rate can only be an underestimate.Establishing always-discarded behavior also requires large base result sets; only two probes were conclusive, both on sets exceeding 246 million rows.Source: §5.1 · pp. 7–8
• Agent experiments cover three vendors versus 27 in the measurement campaign, and model access uses one gateway; vocabulary recovery by brute force makes reported counts lower bounds.Monid is operated by an affiliated organisation, though the static prevalence claim does not depend on it and no vendor influenced the results.Source: §5.1 · p. 8

5.2 Recommendations

• API publishers should encode vocabularies in schemas or use closed lists instead of examples, and distinguish unsupported filters from genuine zero-match results.No HTTP status currently makes those two outcomes distinguishable to automated consumers.Source: §5.2 · p. 9
• Tool-layer authors should expose fewer parameters and validate against schemas with actionable legal-value errors; agent builders can retry without a filter but must not report that result as filtered.One observed model made exactly that unfiltered-retry mistake.Source: §5.2 · p. 9
• Simulated-tool benchmarks cannot test this failure class, while the documentation gap is mechanically detectable and automated repair appears tractable.Simulators generate responses from documentation, which does not describe its own omissions.Source: §5.2 · p. 9

• The failure occurs where constraints are neither disclosed nor machine-checkable, a combination found across 721,320 public parameters.The conclusion says the remedy is one line of schema.Source: §6 · p. 9
• Fully written-out vocabularies are used correctly, whereas vocabularies presented only as examples are not.The conclusion distinguishes disclosure, which guides model selection, from machine-readability, which permits enforcement beyond the model.Source: §6 · p. 9
• Constraint form, rather than vendor identity, predicts silent failure in the paper’s account.Machine-readable constraints are the only form that something other than the model can enforce.Source: §6 · p. 9

A Reproducibility

• Every executed call has a released Monid run identifier, allowing individual measurements to be re-fetched and checked.The full campaign cost under US$8, including US$4.67 in data-API spend and US$1.89 in model inference.Source: A Reproducibility · p. 9
• Caching reduced 815 tool calls to 44 distinct API calls and 88 executions rather than 2,445, making replication after a fix take minutes.Source: A Reproducibility · p. 9
• Because one vendor uses bare HTTP 400 responses for throttling, the study paces calls four seconds apart and repeats every 4xx to confirm deterministic rejection.Skipping confirmation could overstate honest-error rates.Source: A Reproducibility · p. 9

B Perturbation Families

• Each perturbation is deterministically generated from the published schema and the Table 4 synonym lexicon, so released schema dumps allow regeneration.Table 3 provides family examples from the seed endpoint.Source: B Perturbation Families · p. 10, Table 3 · p. 10, Table 4 · p. 10
• Table 3 lists the ten perturbation families.Source: Table 3 · p. 10
• Table 4 gives the full synonym lexicon of substitutions a model might plausibly emit after reading a prose vocabulary rather than a machine-checkable enumeration.Source: Table 4 · p. 10

C Recovered Vocabularies

• The recovered vocabularies are lower bounds: they include only the terms the authors tried.The vocabularies were recovered by brute force.Source: C Recovered Vocabularies · p. 10
• The comparison pairs accepted terms with terms a caller would reasonably use.Source: C Recovered Vocabularies · p. 10
• Reasonable caller terms outside a vendor’s vocabulary are what make the trap consequential.Source: C Recovered Vocabularies · p. 10

D Per-Vendor Outcomes

• Vendors contributing only machine-checkable parameters cannot exhibit silent failure through those parameters.Source: D Per-Vendor Outcomes · p. 10

E Prompt Templates

• Agent trials used one system prompt and task string per case, without few-shot examples, chain-of-thought instructions, or vendor terminology.The tasks contrasted ordinary-English concepts whose obvious rendering was or was not in the vendor vocabulary.Source: E Prompt Templates · pp. 10–11
• A separate model judged answers using the task and answer, but not the tool call.The judge classified outcomes including assertions of absence, hedging, reporting zero as fact, and fabricated numbers.Source: E Prompt Templates · p. 11

F Agent Answers, Verbatim

• A model invented a figure after a zero-row, HTTP 200 call with no error field, despite a question whose true answer was in the millions.Source: F Agent Answers, Verbatim · p. 11
• Another answer estimated 5–7 million engineers and recommended a specialized labor database or government source for precise real-time data.Source: F Agent Answers, Verbatim · pp. 11–12
• Hedging is the majority behavior among the presented answer classifications.Source: F Agent Answers, Verbatim · p. 12
• One answer asserted that zero engineers were found, while also listing possible explanations for the empty search.Source: F Agent Answers, Verbatim · pp. 11–12

G Retry Traces

• None of the retries recovered results; the accepted term for the security cases was “it,” which synonym searches did not reach.Source: G Retry Traces · p. 12
• A retry trace shows a model dropping the filter and receiving 42,619,384 unfiltered rows.Source: Table 12 · p. 12