Clean Scores, Buried Evidence, and Confident Wrong: A Receipt-Based Audit of Frontier Agentic QA

1. Introduction

• The signer bears the consequences while the model does not, so receipts let a person verify the evidence chain rather than rely on confidence.The paper connects this accountability asymmetry to finance and defense-related work.Source: §1 · p. 2
• The audit compares clean and buried conditions in a synthetic finance-and-insurance data room, measuring accuracy, committed errors, forced declarations, tool use, and cost.Its closed corpus has known provenance, addressing a gap in evaluations of open-web or software tasks, multi-hop QA, and long-context retrieval.Source: §1 · p. 2
• Receipts trace each claim to its supporting evidence because confident answers can be wrong or combine accurate numbers with fabricated explanations [11].A receipt records what supports a claim, not why the model produced it.Source: §1 · p. 2
• A wrong answer under buried evidence does not reveal whether retrieval, usable context, interpretation, answer formatting, or evaluation caused the failure.The paper motivates separating these possibilities but does not establish which component dominates.Source: §1 · p. 2

2. Scope and non-claims

• The panel contains one flagship model from each of six closed frontier laboratories; this design does not claim which models are best.The leading open-weight release is excluded as a different category, while Grok-4.6 and Nemotron-3-Ultra-550B are outside the panel.Source: §2 · p. 3
• Table 1 identifies the six-lab panel and reports this paper’s measured room02 accuracy.The paper verifies and freezes its own benchmark results but does not reproduce unverified external benchmark scores or pricing specifications.Source: Table 1 · p. 3, §2 · p. 3
• Hop labels remain corpus-declared task complexity, and the documented Fable 5 fabrication case is an n=1 production observation.The prompt audit found no enumerated substeps in 41 room02 prompts, but does not prove that every declared hop is required.Source: §2 · p. 3

room02 data-room benchmark

• The room02 benchmark has 41 human-designed synthetic finance-and-insurance items, with declared hop counts from two to five and three seeds per model-condition.The data room uses authored documents with known provenance and no real-world proprietary corpus material.Source: §3 · pp. 3–4, room02 data-room benchmark · p. 4
• Clean conditions mount only gold documents, while buried conditions mount the full room including noise; the harness provides read-only files and shell tools.The evaluation elicits an answer, confidence score, and evidence-file list, then measures accuracy, confident-wrong rate, forced declarations, tool calls, spend, and cost per correct answer.Source: room02 data-room benchmark · p. 4

Shallow chart-reading instrument (image input)

• The shallow chart-and-table quiz uses image input and one- and two-hop items to measure reading rather than composition.Its scope includes chart screenshots and newspaper scans; the paper notes that GLM-5.3 was not run because the tested configuration did not accept images.Source: Shallow chart-reading instrument… · p. 4
• 67.6% one-hop and 92.6% two-hop accuracy for Google, while most image-capable models score 100% at one hop.The shallow quiz uses image-rendered charts and tables to measure reading rather than composition; it covers five of the six panel labs and is not a matched comparison with room02.Source: Shallow chart-reading instrument… · p. 4, Table 1b · p. 5

Receipts

• Claim-level receipts trace each statement to a source file so a third party can reconstruct which document supports it without access to the model.The audit treats this traceability as the checkable core of an explanation.Source: Receipts · p. 5

4. Main results: room02 clean vs buried

• The clean-to-buried transition changes accuracy, forced behavior, tool-call volume, and cost in the controlled data-room benchmark.The figures and tables present the paired transitions and per-lab deltas.Source: §4 · p. 5, Table 3 · p. 6, Table 3b · p. 7
• Accuracy columns are frozen historical containment-based scores over 41 items, not exact-match; §8 comparisons use 39 items after two were ruled defective.The restricted population is used only for §8 statistical comparisons.Source: §4 · p. 5
• Forced rows are answered under a closed search budget and graded normally; accuracy is also reported separately for answered and forced rows.The table notes distinguish these conditional accuracy fractions from overall accuracy, forced share, and committed-error percentage.Source: §4 · p. 6
• Table 3 distinguishes total spend from spend per correct or wrong answer; mean tool calls are average invocations per item.The table covers cost and effort in the clean and buried arms.Source: §4 · p. 6, Table 3 · p. 6
• Table 3b reports each lab’s buried-versus-clean arithmetic differences in percentage points or ratios.The deltas are calculated from the arms in Tables 2 and 3.Source: §4 · p. 7, Table 3b · p. 7

5. Declared-hop results

• 100% clean d2 accuracy across all six labs contrasts with 91.7–100% buried d2 accuracy, while d4/d5 are the weakest cells.Source: §5 · p. 8, Table 4 · p. 8
• The prompt audit found no enumerated substeps, demanded intermediates, or handed-over document locators among 41 items, and accuracy is not strictly monotonic by declared depth.Source: §5 · p. 8

Full-panel efficiency and the multi-hop index

• Every lab loses accuracy when evidence is buried, and per-wrong-answer cost is comparable to or exceeds per-correct-answer cost.Source: Figure 4 · p. 10, Figure 5 · p. 10

The clean arm compresses the panel

• The buried depth-4/5 index spans 13.9 points (SD 5.6) versus the clean index’s 6.5 points (SD 3.0), a resolution difference within this sample, not demonstrated field prediction.Source: The clean arm compresses the panel · pp. 9–10, Figure 6 · p. 11
• The authors identify six-hop chains over conflicting sources as a possible distinction between commodity and proprietary answers across finance and defense work.Source: The clean arm compresses the panel · p. 11
• The paper reports no model ranking because pairwise gaps are within sampling noise and the ordering is not stable across scoring instruments.Source: The clean arm compresses the panel · p. 11, Figure 7 · p. 12

• 72% of wrong answers were stated at 80 or above and 9% below 50 across the six labs, but the high-confidence share falls to about 50% under provisional adjudicated scoring.Source: §6 · p. 13, Figure 8 · p. 13
• OpenAI’s Overconfidence Index is +17.0 in the buried condition, while Anthropic’s is +4.6; the index compares mean stated confidence with accuracy in the dominant 90–100 bucket.Source: §6 · p. 13
• In one documented interaction, Fable 5 paired accurate quantitative tables with three unsupported structural claims, and human scrutiny identified the discrepancy.Source: §6 · p. 14
• The production case motivates receipts linking individual statements to supporting evidence; it is a single observation, not an estimate of fabrication rates.Source: §6 · p. 14

7.1 Clean benchmarks understate agentic burden

• Across six labs, burial reduced accuracy by 4.1–17.8 pp, raised tool calls 1.6–3.2×, and increased cost per correct answer, but does not isolate reasoning degradation.Cost per correct answer rose 4.7–7.0× for five labs and 1.6× for Meta; the design also changes what reaches usable context and how the harness behaves.Source: §7.1 · pp. 14–15

7.2 Confidence is not a receipt

• GPT-5.6-sol put all 123 clean answers in the 90–100 confidence bucket yet had a 17.1% confident-wrong rate; citations also require verification against frozen files.A documented field instance paired accurate numeric tables with fabricated structural claims.Source: §7.2 · p. 15

7.3 Depth claims require uncontaminated prompts

• The audit found 0 of 41 prompts enumerated substeps, but hop depth remains corpus-declared, not verified as required or minimal.The prompt audit is necessary but insufficient to show that every declared hop is required or that no shorter path exists.Source: §7.3 · p. 15

7.4 Receipts as an auditing interface

• Receipts map published numbers to frozen files and locators, while multi-metric scoring captures forcing and cost effects missed by accuracy alone.Calibration rows are bucketed or explained as excluded, hop labels have a human-signed prompt audit and gold-chain checklist, and the field incident is preserved as an unedited artifact pack.Source: §7.4 · p. 15

7.5 A receipt the reader cannot follow is not a receipt

• A usable receipt requires a reader-accessible source and a runnable stack whose licenses permit the reader’s institutional use.The paper notes that commercial or governmental evaluators cannot use components whose licenses forbid those uses.Source: §7.5 · p. 16

7.6 Redundancy is not independence

• Across six labs, buried-panel correctness patterns were positively correlated under both scoring approaches, though the measured correlation magnitude varied with the grader.The frozen grader yielded mean pairwise φ = 0.43 over 15 lab pairs and 41 items; provisional adjudication yielded φ = 0.21. The authors do not treat either magnitude as an effect size or infer ensemble effectiveness.Source: §7.6 · p. 16, Figure 9 · p. 17

8. Limitations

• The frozen grader uses normalized containment and numeric tolerances rather than exact match; a strict extended matcher flipped 46 trials wrong-to-right, as a diagnostic rather than a correction.The grader audit found semantically correct answers in unlisted surface forms scored wrong.Source: §8 · pp. 17–18
• In the stratified human audit, 4 of 34 automated verdict raises were unwarranted, while all 39 sampled agreements with the grader were confirmed.Those unwarranted raises were all cases where automated re-adjudication was too lenient.Source: §8 · p. 18
• Two of 41 items were excluded from every arm and condition, so §8 comparisons use 39 while frozen tables and figures retain the 41-item historical record.The excluded items were ruled defective against their own source documents.Source: §8 · p. 18
• Agent-provisional rescoring ranges are not corrected scores; deterministic, byte-identical grading verifies transcription, not grading semantics.The paper keeps human-reviewed corrections distinct from provisional sensitivity analyses.Source: §8 · pp. 18–19
• On 39 items, Alibaba’s clean-to-buried drop was -13.7 pp [-25.6, -1.7] and Google’s was -17.9 [-29.1, -7.7]; the earlier four-of-six count is not carried as a conclusion.The four-of-six result on all 41 items does not survive removal of the defective items.Source: §8 · p. 19
• The gap between the lowest and next lab on the multi-hop index includes zero under every scoring instrument, so the paper reports no model ranking.The comparisons also do not carry grader uncertainty in their sampling intervals.Source: §8 · pp. 17, 19

9. Next stage: interventions on the evidence-to-answer pipeline

• The next evaluation will compare the same open-weight reader with and without a source-linked knowledge-management layer, then compare stock and task-adapted readers under both conditions.It will distinguish retrieval coverage, answer correctness, formatting compliance, and operational failures.Source: §9 · p. 19
• The study will separate development, training, and held-out cases, and held-out answers and trajectories will not inform tuning.Competitive performance by an open-weight model with structured retrieval remains a hypothesis to be tested.Source: §9 · p. 20
• The defense room03 corpus will carry over the finance room’s harness, forced-declaration schema, and receipts discipline, but none of those defense evaluations has been run yet.Its planned tasks include acquisition supply-chain and FOCI audit, foreign-disclosure review, operational-law review, IG record synthesis, and logistics document synthesis.Source: §9 · p. 20
• The answer contract will be pre-registered and scored from the start, with retrieval coverage and answer correctness reported separately.This is intended to keep changes in evidence delivered to a reader distinct from changes in answer grading.Source: §9 · p. 20
• The next-stage question is whether an explicitly auditable evidence pipeline and training informed by observed failures can let a smaller, controllable model compete on demanding document work.The present paper supplies a baseline, not an answer to that question.Source: §9 · p. 20

10. Reproducibility and verification

• All published numbers are generated from a frozen evidence folder, and every ledger row has been mechanically re-verified against the frozen table.The report follows the reproducibility-checklist practice of.Source: §10 · p. 20
• A claim ledger maps each sentence and table cell to a frozen file and line locator.The body text carries no inline source tags.Source: §10 · p. 20
• Tables regenerate byte-identically and the report is hashed, but independent human sign-off of the rows is still pending.The fabrication case is linked to preserved transcript artifacts.Source: §10 · p. 20

Disclosure of AI assistance

• Claude Opus 5 and Claude Fable 5 were used for software development, experimental analysis, manuscript drafting, and revision.Source: Disclosure of AI assistance · p. 21
• AI-generated suggestions and outputs were treated as provisional and checked against source documents, code, or experimental artifacts as appropriate.Source: Disclosure of AI assistance · p. 21
• The author takes responsibility for the study, and the disclosure does not claim every statement received independent human sign-off.The documented AI-assisted interaction in Box 1 is an object of analysis, not independent validation.Source: Disclosure of AI assistance · p. 21

Appendix A room02 prompt audit and gold-chain checklist

• 0 / 41 prompts matched the audit patterns; each was one or two natural sentences with hops implicit and no prescribed retrieval path.Multi-part questions requested answer components without prescribing how to retrieve them.Source: Appendix A · p. 23
• The audit permits “declared hop count” language only with caveats: counts are corpus metadata and chain lengths are design claims, not verified graph distances.The appendix concludes that room02 passes its audit.Source: Appendix A · p. 23
• The gold-chain checklist pairs inferred chains with gold, bridge, and proof documents, and exact-substring searches locate answer spans in gold documents.An ATTENTION span is an answer span not found verbatim in any gold document; it may be paraphrased, computed, or split across documents.Source: Appendix A · p. 23
• 22 of 41 items had at least one ATTENTION span, and a human is instructed to check those items first.Each item has two unsigned checks: whether its chain matches gold documents and whether declared depth is justified.Source: Appendix A · pp. 23–24
• The audit read all 41 prompts and regex-scanned for decomposition cues; all 41 had no enumerated substeps, prescribed intermediates, locators, or template-prior fact types.The chain designation was orchestrator inference from prompt text, not verified against gold.Source: Appendix A · p. 24

Appendix B Full calibration tables, including excluded rows

• Table B1 reports reliability by bucket for all six panel labs in clean and buried conditions; non-numeric confidence rows are skipped, not set to zero.Rows are assigned to confidence buckets or excluded with a reason; rows without numeric confidence are excluded.Source: Table B1 · p. 26, Appendix B · pp. 26–27

C.1 Provenance

• The Fable 5 episode ran from 2026-09-03 23:45 UTC to 2026-09-04 00:44 UTC; its transcript was mechanically extracted under the consult regime.The transcript’s UTC stamps conflict with a provenance label calling the same times ET, which the appendix treats as a labeling error.Source: §C.1 · p. 28

C.2 The three fabricated claims

• Fable 5 made three unsupported structural claims alongside accurate quantitative tables, delivering them with the same confidence and tone as measured numbers.The author detected the issue by challenging inconsistencies and checking source documents; the appendix reports no evidence of context rot.Source: §C.2 · p. 28

C.3 The verbatim admission

• Fable 5 later admitted inventing the script description, the claim that no deep-hop instrument existed, and an unverified reviewer-attack analysis; it distinguished these from real tables.The admission says the fabrications were delivered with the same confidence as the measured numbers.Source: §C.3 · p. 28

D.1 Claim ledger

• 420 of 420 ledger values matched their source cells mechanically, but independent human sign-off has not been performed.The verification also regenerated ten CSV outputs byte-identically; 401 of 420 ledger values are quoted in the manuscript.Source: §D.1 · p. 29

D.2 Table generation and reproducibility

• An independent rerun on 2026-09-07 reproduced the generated outputs byte-identically.Tables 3–5 and the figures read the six labs’ clean and buried arms from the frozen results.Source: §D.2 · p. 30

D.3 Frozen-file manifest

• The wave-2 addendum reports straddled read-rate minus oracle correct-rate of +0.00 for Qwen3.8-Max over 29 cells and -0.03 for Muse-Spark 1.2 over 31 cells.It adds the two models to four reproduced wave-1 rows and sits outside the original paper freeze.Source: §D.3 · p. 30

D.4 Zenodo deposit

• The Zenodo deposit excludes raw result JSONLs and fabrication_case; the paper’s numbers are reproducible from the frozen folder, not the deposit.The raw JSONLs include verbatim filing text, and the evidence-span quotes are redacted in the deposit.Source: §D.4 · p. 31