PaperlayerView exact revision on arXiv

Source-linked AI summary

Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction

Qing Ye, Meng-Hsuan Lin

arXiv:2608.28439v1cs.CLcs.AI

TL;DR

Fidelity-only evaluation can accept answers produced without reading the source, leaving silent failures and externally checkable claims unresolved. The paper uses dispatch-level instrumentation and a two-phase agent design to expose tool-use failures; clean-run results bound false positives, while detection power for wrong answers from successful tool use remains unmeasured.

  • Problem

    Fidelity is necessary but insufficient because matching an extracted value to the given document cannot show that the agent genuinely read it or that externally checkable claims hold.

  • Method

    The paper logs every tool call and uses a two-pass freeze separating document extraction from chamber-based outcome prediction, then derives rule-based failure-attribution and silent-failure checks.

  • Results

    Dispatch records expose silent failures that fidelity-only scoring passes, while fidelity remains high for frontier models and run-to-run stability separates the three evaluated model stacks.

  • Takeaways & Limitations

    The paper recommends instrumenting every tool call and routing suspect runs to human review rather than automatically gating releases on dispatch signals.

  • Takeaways & Limitations

    Detection power is unmeasured for agents that call their tools successfully, satisfy the checked predicate, and still answer from wrong content.

Abstract

from arXiv · show

One model passed our fidelity check without ever opening the datasheet. We found it while qualifying models for an internal extraction service: a structured-output constraint had silently disabled tool use, and the model answered anyway, with fabricated source text. Only the per-tool trace exposed it. Fidelity -- whether an extracted value matches the source -- is the standard measure for agentic document extraction, and it scores that run a success. We therefore log every tool call in an agentic benchmark of 25 hand-curated claims over three components, with 12 more on a fourth, 37 in all. From that dispatch record we build two instruments: a rule-based failure-attribution classifier, and a silent-failure detector whose two rules check only which tools were called, never the extracted value. The detector raises no flag on 207 clean fidelity-passing extractions across three model families, and recovers all 50 planted faults that withhold exactly the tools its rules check. The two results are not symmetric: the first bounds the false-positive rate, the second is recall by construction, and detection power against runs that call their tools and still answer wrongly is unmeasured. A second, independent oracle, a causal chamber that tests whether the datasheet's claims hold under physical measurement, is intentionally partial: it confirms only what the apparatus can exercise, a verifiable envelope of 2 of those 37 claims, and we give a taxonomy of why the rest are not physically gradable. Under a controlled perturbation, fidelity passes throughout while the chamber verdict flips exactly at the measurement uncertainty. Across three deployed model stacks (one destabilised by its serving stack, not by any capability gap) the tool layer buys portability and observability rather than accuracy, and earns its premium only once a document outgrows the context window.

1 Introduction

The paper argues that fidelity alone cannot distinguish genuine document extraction from correct-looking answers produced without reading, nor establish whether externally checkable claims hold. It addresses both gaps with dispatch-level instrumentation, separate physical validation, and deployment analysis.

  • Motivation: Fidelity scoring can pass an answer produced without reading the supplied datasheet.A per-tool trace exposed a model that made zero tool calls while passing fidelity.
  • Benchmark: The benchmark scores 25 hand-curated claims across three components and 12 additional claims on a fourth, 37 in all.Each claim receives separate fidelity and physical reproducibility outcomes.
  • Instrumentation: Dispatch records support a failure-attribution classifier and a silent-failure detector based on navigation and verification-tool presence.The detector flags fidelity-passing runs that bypass navigation or skip verification.
  • Evaluation: The detector produced zero false positives on 207 clean extractions and had by-construction recall of all 50 planted faults.These results concern the detector’s specified tool-bypass and verification-skipped rules.
  • Physical grounding: Physical grounding is partial because the causal chamber can confirm only the claims its instrumentation can exercise.The reported verifiable envelope contains 2 of 37 claims, with a taxonomy for the remainder.
  • Deployment: Across model families, the deployment study examines cost, latency, portability, and context-window limits for agentic extraction.The paper reports that navigation cost tracks the pages needed for a claim rather than document length.

2 Related Work

Prior agentic document systems generally use document-internal correctness oracles, while physical-experiment systems use external validation without physically checking document extraction. This paper combines document extraction with independent physical measurement and per-tool failure attribution.

  • Agentic document understanding: Existing agentic document-understanding systems evaluate correctness with document-internal oracles.The cited systems differ in training, navigation, terminal-tool use, and harness design, but retain document-internal correctness evaluation.
  • Datasheet extraction: ModelGen extracts semiconductor datasheet parameters with a multimodal agentic workflow but uses an LLM to score device-level fitting quality.This paper instead uses independent physical measurement outside any model and scores claims directly.
  • External oracles and failure attribution: Physical-experiment agents validate against external oracles, but the cited work does not verify document extraction through physical measurement.The paper uses Causal Chambers as its physical oracle and positions its contribution as methodology rather than new tools.
  • External oracles and failure attribution: Prior process-diagnosis and trajectory-aware benchmarks operate at per-agent or per-step granularity rather than attributing failures per tool against a fixed rubric.The paper frames per-tool attribution as the remaining methodological distinction.

3 Benchmark Design

The benchmark separates document fidelity from claim reproducibility under matched operating conditions, using a two-phase instrumented agent and fixed tool-based diagnostic rules. Physical verdicts account explicitly for decisive conditions and measurement uncertainty.

  • Benchmark design: The benchmark pairs document-internal fidelity with claim-level reproducibility from an independent causal-chamber measurement.The chamber measures a subset of datasheet quantities under matched conditions where its instrumentation permits.
  • Benchmark design: Fidelity checks the stated value, units, and operating conditions, whereas reproducibility compares chamber measurements with the stated value within spec-appropriate tolerance.The reproducibility verdict is computed by fixed chamber-side code from the curated claim specification and physical measurement.
  • Benchmark design: Reproducibility depends on matching decisive operating conditions such as supply voltage, temperature, sampling rate, filter state, and sample-population size.The benchmark reports which decisive conditions could and could not be matched for each claim.
  • Benchmark design: The chamber treats disagreement smaller than combined measurement uncertainty and claim tolerance as inconclusive rather than a failure.Measurements include instrument and model-form uncertainty, including cross-sensor uncertainty u and claim tolerance t.
  • Claim suite: The 25 claims cover 11 DPS310, 9 Si115x, and 5 ACS70331 claims with component-specific chamber protocols.DPS310 uses cross-sensor agreement, Si115x uses per-channel precision and range checks, and ACS70331 has a distinct test shape.
  • Claim suite: Claims are hand-curated quantitative parameters with explicit values, units, and operating conditions rather than representative samples from full datasheets.The suite is designed to exercise both verdicts.
  • Instrumented agent: The instrumented agent logs inputs, outputs, and timing for every tool call across extraction and chamber phases.Two-pass gating exposes document tools before submission and chamber tools afterward, with submit_extraction freezing the extracted value.
  • Failure attribution: A fixed rubric attributes tool-output errors to the returning tool and selection, condition-omission, and verification-skipped errors to reasoning.This separates tool-fixable failures from failures requiring reasoning changes.

4 Experimental Setup

The experiments compare three model families under a shared agent harness but provider-specific transports, with an agentic engine and single-pass baseline. Three repeats per claim quantify end-to-end variance in deployed configurations.

  • Models and transport: The study evaluates Claude Sonnet 4.6, GPT-5.1, and Qwen3.6-27B under the same harness but different provider transports.Claude and Qwen use an Anthropic-shaped gateway endpoint, while GPT-5.1 uses the OpenAI Responses API.
  • Models and transport: A uniform wire format can silently distort models for which it was not designed, so transport is treated as an experimental factor.The harness log is held constant while serving stacks differ.
  • Engines: Each claim is evaluated with both a full-tool agentic engine and a single-pass baseline using the entire datasheet PDF.Qwen’s baseline renders each PDF page as an image because its vLLM backend cannot ingest PDFs.
  • Repeats: Three repeats per claim per model measure end-to-end variance for each deployed configuration.The study reports mean ± std, with provider-default sampling and pinned turn budgets rather than fixed temperature.
  • Experimental controls: Qwen’s repeats preceded the two-pass freeze, while Claude and GPT-5.1 were run under it after the audit and transport fix.A backend change prevented rerunning Qwen under the freeze; the frontier models’ fidelity remained unchanged when rerun.

5 A Worked Example: A Silent Failure Caught in Development

A provider-specific structured-output constraint blocked tool use, yet the model produced a fluent, fidelity-passing answer with fabricated source text. Dispatch-level traces exposed the failure and identified a uniform tool-based fix.

  • Qwen passed fidelity on every claim with zero tool calls, never reading the datasheet.A guided-decoding token mask forced the answer schema from the first token and blocked tool-call initiation.
  • Removing the output-format parameter and returning structured output through a tool call restored datasheet engagement across providers.The replacement used submit_claim_result, the single-pass predecessor of submit_extraction.
  • Missing navigation or cross-check calls provide a reusable fingerprint for silent, harness-level failures.The benchmark makes answers that were never extracted observable through their dispatch records.

6 Results

The results show that dispatch traces expose silent failures and portability issues that fidelity alone misses, while physical reproducibility is independently testable only within a narrow measurement envelope. Agentic execution adds observability and portability, but accuracy benefits appear primarily when documents exceed context limits.

  • 6.1 Fidelity and run-to-run variance: Claude Sonnet 4.6 and GPT-5.1 each pass all 25 claims in every repeat without engine errors.Qwen3.6-27B passes 23, 19, and 15 claims across repeats; disabling reasoning improves results to 25/24/24 with one engine error.
  • 6.1 Fidelity and run-to-run variance: 12 of 12 off-corpus claims pass for both frontier models, but the result is not evidence of generalisation.The fourth component was already exposed during probing, and zero flags over 12 claims is a zero-event count rather than a precision result.
  • 6.2 Portability: Six portability failures appear across three model families, including two silent failures resolved by using GPT-5.1’s provider-native Responses API.The operational checklist routes warning signals to human review rather than automatic release gates.
  • 6.3 Baseline versus agentic: Across the corpus, baseline and agentic agree on fidelity in 68 of 75 cells, while the tool layer costs 1.2× for Claude and 1.8× for GPT-5.1.The paper reports that agentic execution buys portability and observability rather than accuracy when documents fit one context window.
  • 6.5 Detecting silent failures: The detector flags none of 207 ordinary fidelity-passing cells and recovers all 50 planted faults, but its recall is by construction for rule-aligned faults.The 207 cells are not independent trials, so zero flags bounds the false-positive rate at 4.0% per group or about 11% per claim.
  • 6.5 Detecting silent failures: Tool-presence rules miss wrong-document failures: across 100 Claude and GPT-5.1 probe runs, 8 answered and none was correct.Six answers faithfully transcribed a decoy datasheet, while two were memory answers.
  • 6.6 The reproducibility decomposition: Only 2 of 37 claims lie within the chamber’s definitive reproducibility envelope, leaving 35 claims physically ungradable with available instrumentation.The remaining claims are limited by engagement, traceable-reference, condition-matching, or resolution constraints.
  • 6.6 The reproducibility decomposition: Under controlled perturbation, fidelity passes throughout while reproducibility changes pass → inconclusive → fail exactly at the 0.080 hPa uncertainty band.The real DPS310 pressure is held fixed while the datasheet bound is shrunk through it.

7 Conclusion

The paper concludes that fidelity alone cannot establish trust in agentic extraction. Dispatch records should expose silent failures, while physical verification remains bounded by a stated verifiable envelope.

  • Instrument every tool call to expose silent failures that fidelity-only scoring can pass.The paper recommends routing suspect runs to human review rather than automatically gating releases.

Limitations

The study’s limitations concern detector coverage, physical gradability, deployment-specific scope, reproducibility, and the provenance and calibration of its analyses.

  • Detection power against agents that call tools successfully but answer from wrong content remains unmeasured.Such runs are silent to both the dispatch rules and fidelity scoring.
  • The study omits a human baseline, so the difficulty of the claims for a competent applications engineer remains open.Blind re-derivation bounds required substrings but not human task difficulty.
  • The failure-attribution classifier is supported by only n=27 adjudicated cells, with sensitivity of 3 of 6 and no calibration evidence.The 21 human-clean cells imply a one-sided 95% ceiling near 13% for over-flagging, not proof of zero over-flagging.
  • Reproduction depends on a supplied gateway and therefore extends only up to checkpoint drift.The benchmark, claim file, grading surface, archived runs, and harness are released, but the model itself cannot be shipped.
  • Design choices largely predate discovery of an oracle leak, leaving several thresholds and claim conditions potentially contaminated or unbounded.The detector rules were authored after the fix; the claim set and its conditions remain unbounded.
  • The physical oracle can definitively grade only 2 of 37 claims, leaving most claims outside its verifiable envelope.The fourth component contributes no gradable claim, while the remaining claims fail for engagement, reference, condition, or resolution reasons.
  • The three configurations compare deployed model stacks rather than isolated models, with Qwen instability attributed largely to its serving stack.Transport, input modality, freeze state, and reasoning mode differ alongside model weights.
  • The benchmark covers 25 claims across three components on one gateway plus 12 off-corpus claims, limiting portability conclusions to that operational surface.Cost figures use public list prices.

Ethical Considerations

The benchmark uses public hardware, data, simulators, and manufacturer datasheets without proprietary information. It distinguishes faithful transcription from physical truth rather than treating fidelity as a guarantee.

  • The benchmark is built entirely from public assets and uses or releases no proprietary information.Its inputs include open Causal Chambers resources and publicly available manufacturer datasheets.
  • A fidelity-passing extraction is not a guarantee that the datasheet claim is physically true.The reproducibility axis preserves this distinction for practitioners deploying document-extraction agents.

A Full Cross-Provider Portability Findings

The portability study exposes provider- and backend-specific failures that fidelity alone can miss. Dispatch instrumentation supports deterministic attribution and silent-failure detection, while the grading surface remains partly contaminated by the earlier prompt leak.

  • HTTP 400 errors arose when the gateway translated GPT-5.1 PDF content into a malformed image payload, while its native API accepted PDFs.
  • GPT-5.1 passed fidelity through the passthrough despite discarded reasoning output, making the silent loss visible only in the recorded trace.
  • Qwen required rendered page images because its vLLM backend could not ingest PDF document blocks.
  • A JSON-schema constraint caused Qwen to emit fidelity-passing answers with zero tool calls by suppressing the tool-call opening token.
  • The hosted Qwen backend later regressed on the 3.7 kB tool schema, implicating backend schema handling rather than model capability.
  • The classifier is a deterministic dispatch-record rule set, but Qwen’s 41 non-ok slots included 21 unclassified calls and 14 engine errors, not 41 rubric failures.
  • Human agreement was 24 of 27 adjudicated traces (89%), with three classifier misses and a single-annotator limitation.

D Claim List

The benchmark uses a curated claim file spanning explicit values, units, operating conditions, and chamber bindings. Its physical oracle is intentionally partial: only claims within the apparatus’s verifiable envelope can receive definitive measurement-based verdicts.

  • Each claim records a parameter, expected unit, claim kind, operating conditions, and chamber-side protocol bindings.
  • The frozen set contains 14 typical performance characteristics, 6 operating ranges, and 3 DC-accuracy specifications, with candidates selected from parameter tables.
  • The candidate pool was informal and unreleased, so third parties cannot reproduce the selected claim pool exactly.
  • The 25 claims cite 10 distinct pages, with most extending beyond page 1 into parameter tables.
  • Four gold claims changed after model output, including three ACS70331 corrections where the datasheet overruled the curator.
  • The model-triggered audit cannot exclude curator errors that models happened to reproduce, limiting confidence in unflagged claims.
  • The benchmark is released with offline claims, grading, archived runs, analyses, and an agent harness, but not the served models.
  • The chamber’s bill of materials leaves engagement-only claims without measurements and therefore without a comparison verdict.

F A Fourth Component, Off Corpus

The fourth-component study extends fidelity and detector checks to a 20-page motor-driver datasheet, but remains a weak, fidelity-only null result rather than evidence of generalisation.

  • The off-corpus extension covers 12 claims on a 20-page A4988 stepper-driver datasheet, kept separate from the frozen 25 claims.
  • Table 6 reports navigation calls per claim against each model’s own clean-corpus mean, requiring like-for-like comparisons.
  • The study is weaker than the frozen benchmark because the claims have no declared candidate pool and the document’s numeric needles are often short.
  • Zero detector flags across 12 claims per model is a zero-event count, not a precision result, and its confidence depends on the independence unit.
  • The chamber cannot stage the motor driver, so all 12 off-corpus claims are ungradable and the combined envelope remains 2 of 37.
  • On a later changed tool surface, Claude passed 12 of 12 and GPT-5.1 passed 11 of 11 completed claims, with no detector flags.
  • Navigation counts across runs are not comparable because the tool layer changed its table-of-contents and search behavior.
  • Tool-call counts belong to the model–tool-surface pair, so the surface must be versioned alongside reported numbers.

G Blind Re-Derivation of the Grading Surface

The blind re-derivation exposes substantial sensitivity in the grading surface: notation, floor interpretation, and a guide leak can change verdicts without changing the underlying extraction. It also shows that fidelity-only and locator-based checks leave important silent failures undetected.

  • The unit comparison is structurally unfavorable: the annotator’s needles are stricter, while the paper’s single-character unit needles can match inside compound tokens.The annotator answered 23 of 25 claims; numeric needles match on 15 of 23 and unit needles on 13.
  • One of 25 claims was not strictly blind because the guide used “1.7” as a formatting example, although the annotator recorded “min 1.7”.The record bounds the leak to that claim’s numeric needle.
  • Across 74 stored extractions, self-reported confidence clears the floors by a median of 0.27, and the floor is decisive for exactly one cell.The floors are fixed at 0.7 for 22 claims and 0.6 for three; only one passing cell lies within 0.05 of its floor.
  • 64 of 68 testable cell verdicts move under the re-derived surface, every one from pass to fail.The largest cause is the 1.0 floor, followed by stricter unit needles, range-prefix matching, and two numeric differences.
  • 51 of those 64 flips follow from the 1.0 floor, which rejects every answer the agent has ever given.The causes overlap: 24 cells fail both the floor and a needle, while changing unit needles alone moves 13 cells.
  • The deployed checks miss silent failures: no rule fired on 0 of 14 eligible jobs, while 17 of 84 source-grounding values were not re-locatable without any locator flag.Both reported silent failures were found by human trace inspection, and the locator check flags only locator errors.
Loading 2608.28439v1…