Source-linked AI summary
OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning
Krista Opsahl-Ong, Arnav Singhvi, Jasmine Collins, Ivan Zhou, Cindy Wang, Ashutosh Baheti, Owen Oertell, Jacob Portes, Sam Havens, Erich Elsen, Michael Bendersky, Matei Zaharia, Xing Chen
TL;DR
Enterprise grounded reasoning requires reliable retrieval and analysis across large, heterogeneous document collections, a challenge that existing closed-world or knowledge-focused benchmarks do not fully capture. OfficeQA Pro addresses this gap with a verifiable benchmark built from Treasury Bulletins and evaluates frontier agents, parsing, retrieval, and reasoning choices. Agents remain substantially below reliable enterprise performance, although structured parsing improves results and significant headroom remains.
Problem
Grounded enterprise tasks require retrieving and analyzing relevant materials across large, heterogeneous document collections, which existing closed-world benchmarks do not fully capture.
Method
OfficeQA Pro is a verifiable benchmark built from U.S. Treasury Bulletins that evaluates frontier models and agent architectures across document-intensive grounded-reasoning workflows.
Results
48.1% accuracy is achieved by the strongest frontier agent, increasing up to 56.4% after pre-parsing with Databricks’ ai_parse_document.
Takeaways & Limitations
Document parsing quality, table representation, retrieval strategy, and test-time scaling affect agent performance, but reliable enterprise grounded reasoning remains unsolved.
Takeaways & Limitations
Agents frequently fail to retrieve revised values across documents, often converging on the first numerically plausible instance despite instructions to identify the most recently published figures.
Abstract
from arXiv · showhide
We introduce OfficeQA Pro, a benchmark for evaluating AI agents on grounded, multi-document reasoning over a large and heterogeneous document corpus. The corpus consists of U.S. Treasury Bulletins spanning nearly 100 years, comprising 89,000 pages and over 26 million numerical values. OfficeQA Pro consists of 133 questions that require precise document parsing, retrieval, and analytical reasoning across both unstructured text and tabular data. Frontier LLMs including Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro Preview achieve less than 5% accuracy on OfficeQA Pro when relying on parametric knowledge, and less than 12% with additional access to the web. When provided directly with the document corpus, frontier agents still struggle on over half of questions, scoring 34.1% on average. We find that providing agents with a structured document representation produced by Databricks' ai_parse_document yields a 16.1% average relative performance gain across agents. We conduct additional ablations to study the effects of model selection, table representation, retrieval strategy, and test-time scaling on performance. Despite these improvements, significant headroom remains before agents can be considered reliable at enterprise-grade grounded reasoning.
1 Introduction
OfficeQA Pro targets grounded reasoning in large enterprise document collections, where systems must retrieve relevant materials and analyze them rather than rely on closed-world prompts or parametric knowledge. Despite strong specialized reasoning abilities, frontier models perform poorly on these economically valuable tasks, while the benchmark evaluates parsing, retrieval, and agent design choices.
- Motivation and contribution: Grounded Reasoning requires navigating large, heterogeneous corpora, retrieving relevant materials, and performing grounded analysis.Existing closed-world tasks provide fewer than two artifacts on average and neglect faithful retrieval across large enterprise collections.
- Motivation and contribution: OfficeQA Pro evaluates end-to-end enterprise capabilities across document-intensive workflows using a century-spanning archive of U.S. Treasury Bulletins.The benchmark is designed to proxy economically valuable tasks centered on Grounded Reasoning and provides answers that do not require human-expert grading.
- Motivation and contribution: <50% of questions are answered correctly by state-of-the-art agents with the latest frontier models even when the full corpus is available.The evaluated models include Claude Opus 4.6, GPT 5.4, and Gemini 3.1 Pro Preview.
- Motivation and contribution: Less than 5% of OfficeQA Pro questions are answered correctly by frontier models relying on parametric knowledge alone.This result motivates evaluation settings that require grounded interaction with the document corpus.
- Motivation and contribution: The paper evaluates frontier models and agent architectures, then studies document parsing quality, table representation, search strategies, and test-time scaling.Specialized parsing via Databricks’ ai_parse_document yields a 16.1% average relative performance gain across agents.
2 The OfficeQA Pro Benchmark
OfficeQA Pro is constructed to mirror enterprise document workflows through heterogeneous Treasury Bulletins, multi-step questions, and precise automated evaluation. Its creation process combines annotator verification with agent-assisted quality assurance, while exact-match and error-threshold metrics support reproducible scoring.
- Benchmark design: OfficeQA Pro is designed to capture prose, tabular data, long-term data drift, document noise, and archive scale in enterprise workflows.Its benchmark desiderata also require multi-step retrieval, evidence extraction, computation, and high-precision verifiable evaluation.
- Benchmark composition: U.S. Treasury Bulletins were published monthly from 1939–1982 and quarterly thereafter, forming the benchmark’s multi-decade corpus.Each bulletin contains written analysis, numerical tables, figures, and charts, with nested hierarchies, changing units, and footnotes.
- Benchmark composition: 133 questions comprise OfficeQA Pro, with 113 easier companion questions released for iteration and evaluation of non-frontier systems.Unless otherwise noted, reported results use OfficeQA Pro rather than the combined OfficeQA-Full set.
- Question composition: 62% of questions require data analysis beyond basic arithmetic, 11% require data from at least three bulletins, and 22% require internet search for external values.Questions may require information from one to more than twenty pages, while 3% require visual reasoning over figures, charts, or graphs.
- Creation and verification: Two end-to-end quality-assurance rounds use conflicting agent answers to guide human review of question ambiguity and ground-truth correctness.Initial verification asks a new annotator to reproduce each answer from the source PDF pages, with a third annotator reviewing disagreements.
- Evaluation: Exact match is the default evaluation because each question has one unambiguous answer, while allowable absolute relative-error thresholds accommodate numerical variation.Unless otherwise stated, performance uses a 0.0% allowable absolute relative-error threshold; 99% of answers are numerical.
3 Frontier AI Performance on OfficeQA Pro
Frontier models perform poorly on OfficeQA Pro without grounded document access, while oracle pages and structured parsing substantially improve accuracy, speed, and cost. Nevertheless, extraction and reasoning errors leave substantial performance headroom.
- LLM baselines: Under 3% accuracy at 0.0% allowable absolute relative error occurs for every model in the prompt-only setting.Performance rises to 17–24% at a 5% allowable error threshold, consistent with approximate rather than exact answers.
- LLM baselines: 11.3% accuracy is reached by GPT-5.4 with web search, versus less than 1% in the prompt-only setting.Models still misapply terminology and extract incorrect values despite frequently referencing relevant Treasury sources.
- Oracle document access: 36–57% accuracy is achieved with raw PDFs at 0.0% error, while Databricks-parsed documents improve GPT-5.4 from 57.1% to 65.4% (+8.3pp) and Claude Opus 4.6 from 36.1% to 57.1% (+21.0pp).Gemini 3.1 Pro Preview improves from 52.6% to 56.4% (+3.8pp).
- Remaining limitations: Substantial headroom of approximately 35–44% remains even with oracle pages, web search, and high-quality parsing.Remaining failures include incorrect formulas, unit mismatches, visual-figure interpretation, premature rounding, and wrong retrieved values.
- Oracle document access: 6.0–20.3 absolute percentage-point gains over PDF baselines accompany Databricks-parsed documents, with agents also running 4–9x faster.GPT-5.4 cost drops by approximately 30%, while Opus 4.6 shifts from extraction calls toward token-intensive reasoning.
- Oracle document access: 66.9% accuracy is reached by Claude Opus 4.6 with oracle Databricks-parsed documents, while all agents show their lowest latencies and tool-call counts in this configuration.GPT-5.4 costs $0.33 per sample, approximately 82% below the full-corpus PDF configuration.
4 Custom Agent Experiments
Custom-agent experiments isolate how parser choice and agent design affect OfficeQA Pro performance. Databricks’ ai_parse_document provides the strongest accuracy–cost combination, while further ablations identify model, table, retrieval, and scaling effects.
- Experimental design: Controlled custom-agent studies evaluate parser selection, model choice, table representation, retrieval tools, and test-time compute scaling.The base agent can take up to 200 steps per question and retains the 30 most recent messages in a sliding window.
- PDF parser selection: Databricks’ ai_parse_document, unstructured.io, and Docling are compared using three frontier models and the custom agent.The full-corpus parsing cost is $178 for Databricks’ parser versus $2,670 for unstructured.io.
- PDF parser selection: 50.4% average accuracy is achieved by ai_parse_document, compared with 38.4% for Docling and 31.1% for unstructured.io.The parser also averages $5.29 per sample and is reported as the most cost-effective option.
- Agent design ablations: HTML tables generally improve correctness slightly relative to hierarchical Markdown, while file search combined with contextual-embedding vector search provides the best quality–cost tradeoff.These comparisons are reported across additional agent-design ablations.
- Agent design ablations: 57.1% accuracy is achieved by Claude Opus 4.6 across 10 frontier LLMs, while plurality voting produces modest but consistent gains.Anthropic models dominate the latency Pareto frontier, whereas OpenAI models offer competitive cost efficiency.
5 Remaining Failure Modes
OfficeQA Pro exposes failures in revision-aware retrieval, document parsing, visual interpretation, and quantitative reasoning. These errors can cascade from incorrect evidence extraction into inaccurate calculations.
- Temporal Revision Verification: Revision-aware retrieval fails when agents prematurely select plausible earlier values instead of later revised figures.Repeated revision checks can expand context usage and trigger recursive retrieval loops.
- Parsing Faithfulness: Parsing errors cause 40–50% of baseline-agent failures through misread numbers, corrupted text, and misaligned tables.Even specialized parsing can lose rows, columns, subheaders, descriptions, and footnotes.
- Visual Understanding: Visual figures remain difficult to use because dense charts require fine-grained interpretation and parsed representations can separate visual data from semantic context.These bottlenecks often lead to ungrounded or inaccurate answers.
- Analytical Reasoning: Analytical errors include subtly mismatched values, inappropriate formulas, ignored rounding instructions, and avoidable arithmetic inaccuracies.Examples include using sample instead of population variance and calculating internally rather than delegating to scripts.
6 Related Work
Related benchmarks have expanded frontier-AI evaluation but often underrepresent enterprise workflows requiring retrieval across large document collections. Grounded reasoning connects parsing, retrieval, and quantitative analysis over complex documents.
- Benchmark Scope: HLE and ARC-AGI-2 target academic expertise or abstract intelligence, while GDPval and APEX-Agents simulate professional tasks without fully covering enterprise document retrieval.Closed-world evaluations commonly provide fewer than two artifacts directly in the prompt.
- Grounded Reasoning: Grounded reasoning requires document parsing, information retrieval, and quantitative analysis across complex layouts and heterogeneous document formats.Parsing quality degrades on tables, reading order, and dense formatting, motivating evaluation over OfficeQA Pro’s varied corpus.
7 Discussion & Future Work
OfficeQA Pro is presented as a benchmark for enterprise grounded reasoning, where strong agents and parsing improvements still leave substantial performance and reliability gaps. The discussion identifies parser quality, retrieval, scaling, latency, and broader benchmark coverage as priorities.
- Discussion: OfficeQA Pro evaluates enterprise grounded reasoning tasks with a benchmark designed for AI systems.The benchmark is positioned as an initial step toward broader enterprise evaluations.
- Results: 48.1% accuracy is achieved by the strongest frontier agent, while pre-parsing with ai_parse_document raises performance up to 56.4%.The authors state that substantial headroom remains for reliable reasoning over enterprise documents.
- Key Findings: Parser choice can shift agent accuracy by up to 22 points, while HTML tables, hybrid retrieval, and test-time scaling provide additional but bounded gains.Scaling improvements saturate for higher-performing models, suggesting diminishing returns.
- Future Work: Latency remains substantial: answering over the full PDF corpus averages 23.6 minutes, or 3.9 minutes with state-of-the-art parsing.Future work targets revision-aware search, visual reasoning, and multi-step quantitative reliability.
- Future Work: Future benchmarks should span domains, modalities, and production workflows, with a held-out test set for grounded-reasoning generalization.The proposed expansion includes images, databases, and documents across diverse corpora.
B OfficeQA Full
OfficeQA Full extends OfficeQA Pro with easier questions for evaluating less powerful systems and supporting hillclimbing. Its questions generally require fewer source pages and simpler arithmetic than the core Pro benchmark.
- OfficeQA Full: OfficeQA Full adds 113 easier questions to OfficeQA Pro for hillclimbing and evaluation of less powerful LLM and agent systems.Questions are classified as easy when contemporaneous frontier agents using parsed documents both answered correctly.
- OfficeQA Full: Figure 11 presents the difficulty distribution and capability breakdown for OfficeQA-Full and Pro.The figure distinguishes 113 easy questions from 133 hard questions and summarizes capabilities such as retrieval, data analysis, external search, and visual understanding.
- Question Characteristics: OfficeQA-Full questions tend to require information from fewer unique pages and more basic arithmetic than core OfficeQA Pro questions.The comparison includes basic sums versus more complex data analytics.
- Question Characteristics: Sample questions from both splits illustrate that OfficeQA tasks can require multi-step statistical modeling across multiple documents and datapoints.
C Human Annotator Experiments
The human experiments compare annotators with agents under oracle-page and full-corpus settings, measuring correctness and completion time. Humans generally underperform agents, especially when searching the full corpus, while parsing substantially improves the agents’ relative advantage.
- Experimental setup: 30 questions were evaluated with three annotators under oracle-page and full-corpus conditions, using the same correctness reward function as agent experiments.Oracle pages contained the exact PDF pages needed, while the full-corpus condition linked annotators to the Treasury Bulletin archive.
- Oracle pages: ∼4x longer and ∼28% lower accuracy: with oracle pages, humans were slower and less accurate than agents at 0.0% allowable absolute relative error.Annotators often selected subtly incorrect values, such as Total “Public Debt Securities” instead of Total interest-bearing Public Debt.
- Full corpus: 16.5 points: average human performance decreased when annotators received the full PDF corpus.The comparable average agent decreases were 21.1 points with PDF corpora and 15.5 points with parsed-document corpora.
- Full corpus: ∼2x faster and 44% higher relative accuracy: agents outperformed humans with full-corpus PDF inputs, averaging 15.4 versus 31.4 minutes.With parsed documents, agents were ∼9x faster and had 64% higher relative accuracy.
- Interpretation: Parsing changes the grounding interface: humans can inspect PDF layout and formatting, whereas agents depend on the information preserved by document parsing.When represented correctly, agent performance was substantially better than human performance.
D.2 Model Performance
This section evaluates frontier models using a fixed custom agent configuration across correctness, cost, and latency. Claude Opus 4.6 achieves the highest correctness, while model families occupy different cost–latency trade-offs.
- Model comparison: 57.1% correctness: Claude Opus 4.6 achieved the best performance among the evaluated models.Its average latency remained competitive despite having the highest running cost.
- Model comparison: Anthropic models generally dominate the correctness–latency Pareto frontier, with Gemini 3.1 Pro Preview as an exception.The evaluation reports correctness, cost, latency, and average tool calls for each model.
- Latency: 1.2M versus 2.3M total tokens per question: Claude Opus 4.6 averages fewer tokens than GPT-5.4.Claude models also process input tokens ∼2.7× faster on average than GPT models, at 5,394 versus 1,984 tok/s.
- Cost: OpenAI models remain competitive on the correctness–cost Pareto frontier despite sometimes using more turns and total tokens.The reported pricing estimates do not account for prompt-caching discounts.
D.3 Table Representation
The ablations examine table serialization, retrieval, and test-time scaling in the custom agent. HTML slightly outperforms hierarchical Markdown overall, combined retrieval improves performance, and aggregation benefits lower-performing agents more.
- Table representation: Nested headers are collapsed into single Markdown headers, such as “Sept 30, 1990 > Currency > Total”, to reduce information loss.The dataset’s tables commonly use hierarchical row and column headers that are difficult to represent faithfully in text.
- Table representation: HTML improves correctness in 7 of 11 agents and wins overall by a slight margin over Hierarchical Markdown.Performance differences are generally modest, so table serialization does not dominate overall system behavior.
- Table representation: Model-family preferences vary: GPT and Gemini Pro models favor HTML, while some Claude variants benefit substantially from Hierarchical Markdown.The authors suggest this sensitivity may reflect differences in training-data exposure rather than a universally optimal format.
- Efficiency: Neither table representation consistently dominates latency, tool usage, or cost, and efficiency gains are typically small relative to overall system variance.The comparison used the same base agent with Databricks’ ai_parse_document and file search over the full corpus.
- Search tools: 15% improvement: combining file search with contextual vector search outperformed contextual embeddings alone.The combined configuration achieved the highest accuracy in 2 of 3 models and was 6–13% cheaper than file search alone for GPT-5.4 and Claude Opus 4.6.
- Search tools: 39% relative drop: Gemini 3.1 Pro Preview was especially sensitive when retrieval changed from file search to standard vector search.The authors suggest weaker models may rely more heavily on high-quality retrieval to compensate for downstream limitations.
- Test-time scaling: Plurality-vote gains increase as baseline single-run accuracy decreases, indicating greater benefit from aggregation for higher-variance agents.Stronger agents produce more consistent answers and therefore benefit less from consensus selection.
E.1 Non-Agent Baseline Results
The supplied passages identify the direct-LLM baseline infrastructure and prompting conventions, but do not report baseline accuracy outcomes. The configurations emphasize document access, precision, computation, and a required answer format.
- Environment: The evaluation environment for PDF inputs includes OCR, PDF-processing, image-processing, and table-extraction packages.Listed tools include tesseract, poppler, ghostscript, pymupdf, pdfplumber, camelot, and multiple OCR libraries.
- Configurations: The non-agent baselines share a base system prompt and vary access conditions such as no context, web search, oracle PDFs, and parsed documents.Configuration-specific prompt modifications are listed for the baseline evaluations.
- Prompting: The prompts instruct models to use Treasury documents, perform required computations, and use maximum precision unless the question specifies otherwise.The agent prompt also directs models to use file-system search over the corpus and web search when external information is needed.
- Prompting: The required completion format asks models to retain final reasoning before returning only the answer value inside XML tags.A missing <FINAL_ANSWER> tag is defined as a failure condition.