PaperlayerView exact revision on arXiv

Source-linked AI summary

OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning

Krista Opsahl-Ong, Arnav Singhvi, Jasmine Collins, Ivan Zhou, Cindy Wang, Ashutosh Baheti, Owen Oertell, Jacob Portes, Sam Havens, Erich Elsen, Michael Bendersky, Matei Zaharia, Xing Chen

arXiv:2603.08655v1cs.AIcs.CLcs.IR

TL;DR

Evaluations that supply documents directly in prompts omit the retrieval required for grounded analysis across large enterprise collections. OfficeQA Pro tests end-to-end retrieval and reasoning over a century of Treasury Bulletins, where the strongest frontier agent achieves 48.1% accuracy.

  • Problem

    Evaluations that supply documents directly in prompts omit faithful retrieval across large enterprise collections required for real-world grounded analysis.

    Source: §1 · p. 2

  • Method

    OfficeQA Pro benchmarks end-to-end retrieval and multi-step reasoning over a century of Treasury Bulletins, with unambiguous answers designed for automated evaluation.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    Frontier agents struggle even with full corpus access, with the strongest achieving 48.1% accuracy on OfficeQA Pro.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    Document parsing quality is a critical bottleneck because parsing errors can propagate through retrieval and downstream computation.

    Source: §7 · p. 12

    Agents frequently fail to retrieve revised values, even when explicitly instructed to identify the most recently published figures.

    Source: §5.1 · p. 10

Abstract

from arXiv · show

We introduce OfficeQA Pro, a benchmark for evaluating AI agents on grounded, multi-document reasoning over a large and heterogeneous document corpus. The corpus consists of U.S. Treasury Bulletins spanning nearly 100 years, comprising 89,000 pages and over 26 million numerical values. OfficeQA Pro consists of 133 questions that require precise document parsing, retrieval, and analytical reasoning across both unstructured text and tabular data. Frontier LLMs including Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro Preview achieve less than 5% accuracy on OfficeQA Pro when relying on parametric knowledge, and less than 12% with additional access to the web. When provided directly with the document corpus, frontier agents still struggle on over half of questions, scoring 34.1% on average. We find that providing agents with a structured document representation produced by Databricks' ai_parse_document yields a 16.1% average relative performance gain across agents. We conduct additional ablations to study the effects of model selection, table representation, retrieval strategy, and test-time scaling on performance. Despite these improvements, significant headroom remains before agents can be considered reliable at enterprise-grade grounded reasoning.

1 Introduction

OfficeQA Pro evaluates end-to-end retrieval and grounded analysis over a century of U.S. Treasury Bulletins, with answers verifiable without human-expert grading. Latest frontier agents average 34.1% accuracy, and specialized document parsing yields a 16.1% average relative performance gain.

1 Introduction

  • 34.1% average accuracy is achieved by agents using latest frontier models on OfficeQA Pro; none surpass 50% despite full-corpus access.The latest models evaluated are Claude Opus 4.6, GPT 5.4, and Gemini 3.1 Pro Preview.Source: Figure 1 · p. 1, §1 · p. 2
  • GDPval (Patwardhan et al., 2025) supplies fewer than two artifacts per task on average, omitting large-corpus retrieval and requiring onerous human-expert grading.The authors define Grounded Reasoning as navigating heterogeneous document collections, retrieving relevant materials, and performing grounded analysis.Source: §1 · pp. 1–2
  • OfficeQA Pro evaluates end-to-end retrieval and grounded analysis across a century of U.S. Treasury Bulletins, with verifiable answers requiring no human-expert grading.The benchmark is designed to proxy economically valuable, document-intensive enterprise workflows.Source: §1 · p. 2
  • Less than 5% accuracy is achieved by frontier models answering OfficeQA Pro questions using only parametric knowledge.Strong performance on specialized Olympiad-style reasoning tasks does not translate to the economically valuable tasks proxied here.Source: §1 · p. 2
  • 16.1% average relative performance gain across agents comes from specialized document parsing with Databricks’ ai_parse_document.The study also compares frontier models and agent architectures with human annotators and examines table representation, search strategies, and test-time scaling.Source: §1 · p. 2

2 The OfficeQA Pro Benchmark

OfficeQA Pro evaluates grounded, multi-step reasoning through 133 questions over U.S. Treasury Bulletins. Its design combines heterogeneous documents, human-verified questions, difficulty filtering, and strict answer evaluation.

2.1 Dataset Desiderata

  • The benchmark targets enterprise document complexity, multi-step retrieval and computation beyond parametric knowledge, and high-precision evaluation against unambiguous ground truth.Its design explicitly includes prose and tables, long-term data drift, archival noise, and the need to interpret evidence in context.Source: §2.1 · p. 2

2.2 Benchmark Composition

  • Treasury Bulletins combine complex tables, changing units, extensive footnotes, revised statistics, and variable document quality, requiring contextual interpretation and longitudinal reconciliation.The corpus transitions from scans to digitally native PDFs in 1996. Embedded text layers are removed to ensure fair assessment of document interpretability.Source: §2.2 · p. 2
  • 133 questions constitute OfficeQA Pro, grounded in U.S. Treasury Bulletins published monthly from 1939–1982 and quarterly thereafter.A companion set contains 113 easier questions; together, Pro and Easy form OfficeQA-Full. Unless otherwise noted, reported results use OfficeQA Pro.Source: §2.2 · pp. 2–3
  • 62% of questions require analysis beyond basic arithmetic, 22% require internet search, 11% span at least three bulletins, and 3% require visual reasoning.Questions draw on one to over twenty pages, and each has a single unambiguous answer. With relevant financial or statistical background knowledge, they should be solvable by a numerically literate college graduate.Source: §2.2 · p. 3

2.3 Creation Process

  • Human-authored questions underwent independent answer reproduction and two rounds of agent-assisted quality assurance to establish realistic, unambiguous, verifiable tasks.SuperAnnotate and Turing helped expand seed questions, with USAFacts reviewing their analytical realism; disagreements during independent reproduction triggered third-annotator review. Subsequent reviews of conflicting agent answers distinguished agent failures from ambiguous questions or incorrect ground truth, prompting revisions or corrections.Source: §2.3 · pp. 3–4
  • 246 questions survived parametric-knowledge filtering; excluding questions both development agents answered correctly left 133 questions in OfficeQA Pro.Filtering used Claude Opus 4.5 and GPT-5.1; newer Claude Opus 4.6 and GPT-5.4 can answer some retained questions parametrically. Development agents used Databricks Parsed documents, and questions both answered correctly became Easy.Source: §2.3 · p. 4

2.4 Evaluation

  • 0.0% allowable absolute relative error is the default evaluation threshold; 99% of benchmark answers are numerical.Evaluation also reports accuracy at other error thresholds and normalizes punctuation, mathematical symbols, and common abbreviations. Complex answers require matching text and numerical components, while non-numerical answers use fuzzy text matching.Source: §2.4 · p. 5

3 Frontier AI Performance on OfficeQA Pro

OfficeQA Pro evaluates single-call LLMs and agents capable of autonomous tool use and multistep reasoning. Oracle pages and parsed documents improve performance, with oracle parsed inputs producing the best agent results, but substantial reasoning and precision failures remain.

3.1 LLM Baselines

  • 50.2% average relative accuracy improvement comes from replacing oracle PDF pages with Databricks-parsed pages for LLMs at 0.0% allowable absolute relative error.GPT-5.4 improves from 57.1% to 65.4% (+8.3pp), Claude Opus 4.6 from 36.1% to 57.1% (+21.0pp), and Gemini 3.1 Pro Preview from 52.6% to 56.4% (+3.8pp). Databricks’ ai_parse_document supplies a structured representation encoding text, tables, and other document information.Source: §3.1 · pp. 5–7, Figure 5 · p. 6
  • 11.3% accuracy with web search improves GPT-5.4 over its less-than-1% prompt-only result, but single-call LLM performance remains limited.Every model scores under 3% at 0.0% allowable absolute relative error with prompts alone, rising to 17–24% at 5% allowable error. Prompt-only models refuse 10–15% of questions and sometimes misremember data or use values from the wrong context.Source: §3.1 · pp. 5–6
  • Approximately 80%, 15%, and 2% of web-enabled runs exhaust generation tokens without final answers for Opus 4.6, Gemini 3.1 Pro Preview, and GPT-5.4, respectively.Models still extract or apply data incorrectly despite referencing sources mentioning Treasury Bulletin data. None references the publicly available GitHub dataset containing the exact questions, answers, and ground-truth source documents.Source: §3.1 · p. 6
  • Approximately 15% worse GPT-5.4 performance on 1930s–1970s documents than 1980s–2020s PDFs highlights persistent extraction difficulties; Opus and Gemini show approximately 5% gaps.Raw-PDF errors include misreading nested tables and older scans, while reasoning failures include incorrect formulas, misaligned units, and misinterpreted figures. Approximately 35–44% headroom remains across models even with oracle pages, web search, and high-quality parsing.Source: §3.1 · p. 7

3.2 Agent Baselines

  • 66.9% accuracy makes Claude Opus 4.6 the highest-scoring agent when provided oracle Databricks-parsed documents, the best-performing configuration for all agents.Relative to oracle PDFs, correctness improves by 6–11 percentage points for the Opus 4.6 and GPT-5.4 agents. This configuration minimizes latency and tool calls; GPT-5.4 costs $0.33 per sample, approximately 82% cheaper than its full-corpus PDF configuration.Source: Table 1 · p. 7, §3.2 · p. 8
  • 6.0–20.3 percentage-point correctness gains and 4–9x faster execution result from giving agents Databricks-parsed documents rather than the PDF corpus.Full-corpus PDF accuracy is 48.1% for Claude Opus 4.6, 36.1% for GPT-5.4 High, and 18.1% for Gemini 3.1 Pro Preview. GPT-5.4 costs fall by approximately 30%, while Opus costs remain comparable as extraction calls give way to fewer, more token-intensive reasoning steps.Source: Table 1 · p. 7, §3.2 · pp. 7–8
  • 13–21 percentage-point accuracy gains follow giving agents oracle PDF pages instead of the full PDF corpus, isolating a substantial retrieval bottleneck.Average latency decreases approximately 76%, and costs fall 74–88% across agents. Remaining runtime is dominated by PDF extraction overhead, including repeated file reads and OCR attempts.Source: §3.2 · pp. 7–8
  • Remaining agent errors predominantly involve answer precision and incorrect statistical formulas, alongside wrong table or web values and difficulty with figures omitted from parsed representations.Premature rounding is one precision failure; selecting the wrong row-column value is another retrieval failure. Encoding visual figures is left for future work.Source: §3.2 · p. 8

4 Custom Agent Experiments

Controlled custom-agent experiments compare parser configurations and ablate agent-design choices. ai_parse_document yields the highest accuracy and lowest per-sample agent cost among the evaluated parsers.

4.1 Base Agent Design

  • 200 steps per question and a 30-message sliding window bound the custom agent used for controlled evaluation of individual design decisions.The backend LLM chooses tools, reasons about next steps, and produces answers, with reminders of remaining steps. Default tools include web search, persistent Python execution sandboxed against bulk corpus scanning, and file search restricted by working directory and output size.Source: §4.1 · p. 9, §4 · p. 9

4.2 PDF Parser Selection

  • 50.4% average accuracy at the 0.0% error threshold with ai_parse_document exceeds Docling (38.4%) and unstructured.io (31.1%) across three frontier-model configurations.Average agent cost is $5.29 per sample, 35% below the next-cheapest option, Docling ($8.18). Experiments use GPT 5.4, Claude Opus 4.6, and Gemini 3.1 Pro Preview; ai_parse_document is retained for subsequent ablations.Source: Figure 7 · p. 9, §4.2 · p. 10
  • $178 for full-corpus parsing with ai_parse_document compares with $2,670 for unstructured.io; the third parser, Docling, uses RapidOCR and TableFormer.unstructured.io uses its proprietary high-resolution parser. Parser outputs are simplified by extracting page-element text and concatenating it in reading order into one .txt file per PDF.Source: §4.2 · pp. 9–10
  • 57.1% accuracy makes Claude Opus 4.6 the highest-performing model among ten frontier LLMs evaluated in additional ablations.Anthropic models dominate the latency Pareto frontier, while OpenAI models offer competitive cost efficiency through lower per-token pricing.Source: §4.2 · p. 10
  • Combining file search with contextual-embedding vector search provides the best quality–cost tradeoff among evaluated search-tool configurations.These retrieval experiments are reported in Appendix D.4.Source: §4.2 · p. 10
  • HTML tables generally improve correctness slightly over hierarchical Markdown, while test-time scaling through plurality voting yields modest but consistent gains.The table-representation and plurality-voting ablations appear in Appendices D.3 and D.5, respectively.Source: §4.2 · p. 10

5 Remaining Failure Modes

Remaining OfficeQA Pro failures span temporal revision verification, parsing faithfulness, visual understanding, and analytical reasoning.

5.1 Temporal Revision Verification

  • Agents frequently accept the first numerically plausible value rather than verifying revisions, even when explicitly asked for the most recently published figures.Statistics recur across bulletins with revised values, while reporting conventions change their placement and aggregation. Enforcing revision checks can trigger retrieval loops that saturate context, erase validated intermediate results, and produce approximations and cascading errors when step limits are reached.Source: §5.1 · p. 10

5.2 Parsing Faithfulness

  • 40–50% is the parsing-error failure rate for baseline agents using original PDFs.Misread numbers, corrupted text, misaligned tables, shifted or missing rows and columns, and lost metadata impair retrieval and calculation. Even best-performing agents using Databricks’ ai_parse_document remain affected by parsing errors.Source: §5.2 · p. 10

5.3 Visual Understanding

  • Agents struggle to interpret dense financial charts directly, while lossy embeddings of pre-parsed documents disconnect visual data from its semantic context.Direct image analysis lacks the granular resolution needed to map trend lines precisely. With pre-parsed documents, the resulting retrieval bottleneck impairs visual reasoning and leads to ungrounded or inaccurate answers.Source: §5.3 · p. 11

5.4 Analytical Reasoning

  • Agents select subtly incorrect quantities, apply contextually inappropriate formulas, disregard calculation requirements, and sometimes calculate internally rather than using scripts.Examples include mismatched statistical definitions, imprecise external CPI values, and sample variance substituted for population variance. Intermediate rounding can cause cascading imprecision, while internal calculations introduce avoidable numerical inaccuracies.Source: §5.4 · p. 11

6 Related Work

OfficeQA Pro connects enterprise-workflow evaluation with grounded reasoning over heterogeneous documents. It unifies document parsing, information retrieval, and quantitative analysis within a final verifiable answer.

6 Related Work

  • Humanity’s Last Exam (Phan et al., 2025) and ARC-AGI-2 (Chollet et al., 2026) test expert academics and abstract intelligence but offer limited evidence of real-world value.These benchmarks largely remain decoupled from enterprise workflows.Source: §6 · p. 12
  • Professional-environment benchmarks GDPVal (Patwardhan et al., 2025) and APEX-Agents evaluate deliverable production and long-horizon workspace tasks, respectively.GDPVal tasks span key economic sectors and require reports or spreadsheets; APEX-Agents tasks include updating financial models and reviewing legal documents.Source: §6 · p. 12
  • OfficeQA Pro evaluates end-to-end parsing, retrieval, and computation over historical U.S. Treasury Bulletins, targeting professionally relevant workflows with verifiable answers.Its large, heterogeneous, publicly available document corpus emulates complex enterprise data.Source: §6 · p. 12
  • OmniDocBench (Ouyang et al., 2025) demonstrates extraction degradation on complex PDF layouts, relevant to OfficeQA Pro’s varied document formats.Challenges include tables, reading order, and dense formatting; OfficeQA Pro spans pre-1996 scanned documents and modern digital PDFs.Source: §6 · p. 12
  • BRIGHT (Su et al., 2025) and BrowseComp-Plus (Chen et al., 2025) expose semantic and keyword matching failures when retrieval requires reasoning about intent.Similarly, OfficeQA Pro requires agents to actively identify relevant documents and sections.Source: §6 · p. 12
  • FinanceBench (Islam et al., 2023) evaluates faithful financial extraction and computation, while LongBenchv2 (Bai et al., 2025) highlights persistent long-document reasoning difficulties.OfficeQA Pro likewise requires reliable quantitative analysis even after the correct evidence has been retrieved.Source: §6 · p. 12

7 Discussion & Future Work · A Contributions · A.1 Authors

OfficeQA Pro evaluates enterprise grounded reasoning: the strongest frontier agent achieves 48.1% accuracy, while corpus pre-parsing improves performance up to 56.4%. Substantial headroom remains in document parsing, analytical reliability, and latency, motivating further system improvements and broader evaluations.

7 Discussion & Future Work

  • 56.4% accuracy is reached with Databricks’ ai_parse_document pre-parsing, compared with the strongest frontier agent’s 48.1% accuracy.OfficeQA Pro evaluates AI systems on enterprise grounded reasoning tasks; substantial headroom remains for reliable reasoning over enterprise documents.Source: §7 · p. 12
  • 22 points is the maximum agent-accuracy shift from parser choice; even state-of-the-art parsing errors can propagate through retrieval and downstream computation.Document parsing quality is identified as a critical, often underappreciated bottleneck.Source: §7 · p. 12
  • HTML tables marginally improve correctness over markdown, while combining file-based and vector-based retrieval offers a stronger quality–efficiency tradeoff than either alone.Test-time scaling provides additional gains, but improvements saturate for higher-performing models, suggesting diminishing returns as reasoning becomes more consistent.Source: §7 · p. 12
  • AI agents consistently outperform human annotators in both speed and accuracy, despite struggling to achieve high performance.The human comparison is reported in Appendix C.Source: §7 · p. 12
  • 23.6 minutes per question is the frontier-agent average over the full PDF corpus, versus 3.9 minutes with state-of-the-art document parsing.Latency remains a major, underexplored challenge beyond correctness.Source: §7 · p. 12
  • Failure-mode analysis motivates more exhaustive, revision-aware search, stronger visual reasoning over charts and figures, and more reliable multi-step quantitative reasoning.Substantial headroom remains for both LLM baselines and AI agents.Source: §7 · p. 12
  • Future benchmarks should span more domains, modalities, and production workflows; the authors plan a held-out test set measuring grounded-reasoning generalization.OfficeQA Pro is positioned as an initial step toward broader enterprise evaluations incorporating images, databases, and documents across diverse corpora.Source: §7 · p. 12

A.1 Authors

  • Krista Opsahl-Ong and Arnav Singhvi are credited with equal contribution, with their order alphabetical.The paper lists 13 authors.Source: §A.1 · p. 14

B OfficeQA Full

OfficeQA Full extends OfficeQA Pro with easier questions for hillclimbing and evaluating less powerful systems. These questions generally require fewer distinct pages and simpler arithmetic than the core Pro questions.

B OfficeQA Full

  • 113 additional easier questions extend OfficeQA Pro’s 133-question core to support hillclimbing and evaluation of less powerful LLMs and agents.Figure 11 compares difficulty and capability distributions, covering data analysis, retrieval from >3 distinct pages, external knowledge search, and visual understanding.Source: B OfficeQA Full · p. 14, Figure 11 · p. 14
  • Questions were classified as “Easy” when both frontier agents, using GPT 5.1 and Claude Opus 4.5 with Databricks-parsed documents, answered correctly.The classification used state-of-the-art frontier agents at the time.Source: B OfficeQA Full · p. 14
  • Easier OfficeQA Full questions generally draw on fewer distinct pages and favor basic arithmetic over complex data analytics.Figure 12 contrasts Full and Pro examples; benchmark questions typically require multi-step analytical reasoning, such as statistical modeling across documents and datapoints.Source: B OfficeQA Full · p. 14, Figure 12 · p. 15

C Human Annotator Experiments · D Custom Agent Ablation Studies · D.1 Base Agent Design

On a 30-question OfficeQA-Full subset, agents outperformed human annotators in accuracy and speed with both oracle pages and the full corpus. The ablations use a custom agent with Databricks-parsed documents to study design choices.

C Human Annotator Experiments

  • 30 randomly selected questions were evaluated by three benchmark-familiar annotators using oracle PDF pages with optional web search or the searchable full corpus.Full-corpus trials preceded oracle trials, and annotators never answered questions they created or reviewed. Humans received agent instructions on numerical precision and revised values and used the same correctness reward function.Source: C Human Annotator Experiments · p. 15, Table 2 · p. 15
  • 44% higher relative accuracy and ∼2x faster answers characterized full-corpus PDF agents versus humans; parsed-document agents achieved 64% higher relative accuracy and were ∼9x faster.Average completion times with PDF inputs were 15.4 minutes for agents versus 31.4 minutes for humans.Source: Table 2 · p. 15, C Human Annotator Experiments · p. 16
  • ∼28% lower accuracy relative to average agents and ∼4x longer completion times characterized humans given oracle pages at 0.0% allowable absolute relative error.Even with oracle PDFs, humans most often failed by selecting values subtly different from those requested.Source: C Human Annotator Experiments · p. 16
  • Human instruction-following errors changed answer scales, while transcription errors introduced mistakes not observed in programmatically operating agents.Errors included incorrect units, percentages instead of decimals, and mistyped intermediate or final values.Source: C Human Annotator Experiments · p. 16
  • Agents exhibited raw-PDF parsing failures and content-policy refusals absent from humans; both failure modes decreased significantly with parsed documents.Humans directly interpret page layout and visual content, whereas agents depend on a parsing translation layer whose losses persist through subsequent reasoning.Source: C Human Annotator Experiments · p. 16
  • 16.5 points of performance were lost by humans moving to the full corpus, versus 21.1 points for PDF agents and 15.5 points for parsed-document agents.The human decrease was broadly comparable to the agent decreases.Source: C Human Annotator Experiments · p. 16

D Custom Agent Ablation Studies

  • The custom-agent ablations examine how model selection, table representation, search tools, and test-time scaling affect end-to-end performance.These experiments continue the ablation studies from Section 4.Source: D Custom Agent Ablation Studies · p. 17

D.1 Base Agent Design

  • The base agent uses Databricks-parsed documents, web search, a Python REPL, and sandboxed fs_search and fs_read tools to support systematic ablations.The file tools provide capabilities similar to grep, sed, cat, and ls. Databricks parsing was selected because it performed best among the parsers evaluated in Section 4.2.Source: §D.1 · p. 17

D.2 Model Performance

Claude Opus 4.6 achieved the highest correctness but also the highest cost. Anthropic models generally led the correctness–latency frontier, while OpenAI models remained competitive on correctness versus cost.

D.2 Model Performance

  • 57.1% correctness made Claude Opus 4.6 the best-performing model tested, although it was also the most expensive.The comparison used the custom agent with Databricks’ ai_parse_document and file search at a 0.0% error threshold. Its average latency was lower than about half of the tested agents.Source: Figure 14 · p. 17, Table 4 · p. 18, §D.2 · p. 18
  • Model comparisons held the custom agent configuration fixed, using high reasoning for Anthropic and OpenAI models and default settings for Google models.Anthropic defaults already used high reasoning, whereas OpenAI reasoning was explicitly raised for comparability. Google models did not support reasoning settings.Source: §D.2 · pp. 17–18
  • Anthropic models dominated the correctness–latency Pareto frontier, with Gemini 3.1 Pro Preview the exception.Lower cumulative token usage appears to contribute: Claude Opus 4.6 averaged 1.2M tokens per question versus GPT 5.4’s 2.3M. This pattern did not extend to Claude Haiku 4.5, which averaged 2.2M tokens.Source: §D.2 · p. 18
  • Approximately 2.7× faster estimated input-token processing favored Claude over GPT models, averaging 5,394 versus 1,984 tokens per second.These rates divide average input tokens per rollout by the average total time required to answer a question, rather than measuring input processing in isolation.Source: §D.2 · p. 18
  • GPT-5.1-High and GPT-5.4-High remained competitive on the correctness–cost Pareto frontier because lower input-token prices offset higher token usage.GPT-5.4 input pricing was $2.50/MTok, versus $5.00 for Claude Opus 4.6 and $3.00 for Claude Sonnet 4.5; repeated retrieval, tool usage, and context accumulation magnify these differences. Estimates exclude prompt-caching discounts, although the authors expect directionally similar findings because Claude cached-input pricing remains approximately twice GPT pricing.Source: §D.2 · p. 18

D.3 Table Representation … E Frontier AI Baseline Experiment Details

Combining file search with contextual vector search improved performance; table-format preferences varied by model, and plurality voting benefited lower-accuracy agents more.

D.3 Table Representation

  • Hierarchical Markdown collapses nested headers into single header strings to reduce information loss when representing the corpus’s complicated tables in token-efficient Markdown.Unlike HTML, Markdown does not natively support nested headers; a collapsed header can read “Sept 30, 1990 > Currency > Total”.Source: §D.3 · p. 19
  • All GPT models matched or improved correctness with HTML, both Gemini Pro variants favored HTML, and Claude models showed mixed table-format preferences.Claude Opus 4.5 and Claude Sonnet 4.5 benefited substantially from Hierarchical Markdown, indicating model-dependent sensitivity rather than a universally optimal format.Source: §D.3 · pp. 19–20
  • Neither HTML nor Hierarchical Markdown consistently dominated latency, tool usage, or cost; efficiency gains were model-specific and typically small relative to overall system variance.Source: §D.3 · p. 20
  • 7 out of 11 agents achieved higher correctness with HTML than Hierarchical Markdown, although overall performance differences were generally modest.The comparison used the custom agent with ai_parse_document and file search over the full corpus, evaluated at a 0.0% error threshold.Source: §D.3 · p. 19, Table 5 · p. 20

D.4 Search Tools

  • 21% average performance improvement over standard vector search came from contextual embeddings that appended document, date, page, and table or section metadata to chunks.Contextual vector search reduced tool calls by approximately 44%, latency by approximately 38%, and cost by approximately 44%, placing performance within 12% of file search on average. Standard vector search averaged a 27% relative performance drop versus file search, with chunks often disconnected from essential context such as table headers.Source: §D.4 · pp. 20–21
  • 39% relative performance loss from file search to standard vector search made Gemini 3.1 Pro Preview the model most sensitive to retrieval choice.The authors suggest that its lower performance ceiling may increase dependence on high-quality retrieval to compensate for downstream reasoning, tool-use, or synthesis limitations.Source: §D.4 · p. 21
  • 15% higher performance than contextual vector search alone came from combining it with file search, achieving the highest accuracy in two of three models.The combined configuration was 6–13% cheaper than file search alone for GPT-5.4 and Claude Opus 4.6. Experiments evaluated these models and Gemini 3.1 Pro Preview using HTML tables.Source: §D.4 · pp. 20–21, Table 6 · p. 21

D.5 Test-Time Scaling

  • Plurality voting produced larger improvements for agents with lower average single-run accuracy, while stronger agents benefited less from consensus selection.Using combined file and contextual vector search, the three frontier agents were each run four times; answer selection was compared through plurality voting, average performance, and oracle selection.Source: §D.5 · pp. 21–22

E.1 Non-Agent Baseline Results … E.5 Agent Prompt

These appendices document baseline correctness reporting, cost assumptions, PDF-processing libraries, and prompts for non-agent and agent configurations. The prompts specify evidence access, calculation precision, and completion requirements.

E.1 Non-Agent Baseline Results

  • Table 7 summarizes LLM baseline correctness across evaluated configurations.The supplied caption contains no scores, so it does not establish numerical comparisons or a winner.Source: Table 7 · p. 22

E.2 Cost Assumptions

  • Cost calculations use token-pricing assumptions expressed in USD per million tokens and sourced from provider pricing pages as of March 2026.Table 8 identifies these assumptions, but the supplied caption contains no individual prices.Source: Table 8 · p. 22

E.3 Preinstalled Libraries for Baseline Agent SDKs

  • PDF-equipped baseline-agent sandboxes include system and Python tools for OCR, PDF processing, and table extraction.Preinstalled packages include Tesseract, Poppler utilities, OCRmyPDF, PyMuPDF, pdfplumber, Camelot, and PaddleOCR.Source: §E.3 · pp. 22–23

E.4 Non-Agent (Direct LLM) Prompt

  • The shared non-agent base prompt requires a computed or reasoned answer and maximum calculation precision unless the question specifies otherwise.Agent prompts likewise require computation or reasoning and maximum precision unless otherwise specified.Source: §E.4 · p. 23, §E.5 · pp. 23–24
  • Non-agent completions must present reasoning and sources before the answer value in XML tags; omitting <FINAL_ANSWER> makes the response incomplete.The prompt explicitly treats the missing tag as task failure.Source: §E.4 · p. 23
  • Non-agent configurations vary question-only input, web access, oracle PDF images, parsed document content, and combined PDF-plus-web access.No Context uses the unchanged base prompt; Web Search adds search permission; Oracle PDF supplies pages as base64-encoded images; PDF + Web Search combines both. Oracle Parsed places the base prompt, extracted content, and question in one user message without a system message.Source: §E.4 · p. 23

E.5 Agent Prompt

  • Agent prompts permit filesystem and web search, discourage reading entire documents, and prioritize the latest metric revision unless the question specifies otherwise.The same prompt serves Claude Agent SDK, OpenAI Codex SDK, Gemini CLI, and the custom agent setup. Revision exceptions apply when a question specifies an exact date or document.Source: §E.5 · p. 23

Related papers

Loading 2603.08655v1…