Source-linked AI summary
GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents
Suhaas Garre, Emily Ritchie, Sushant Mehta, Edwin Chen
TL;DR
Standard visual-reasoning benchmarks are a poor guide to professional PDF workflows, where answers may depend on distant evidence and document structure. GDP.pdf evaluates 100 practitioner-authored questions over original PDFs in ten domains, with expert rubrics, capability tags, and items screened for frontier-model failures. The reported pass-rate range was 2–30.7%, and no frontier model passed even a third of the items.
Problem
Standard visual-reasoning benchmarks are a poor guide to professional PDF workflows, where answers may depend on distant evidence and document structure.
Source: §1 · p. 1Method
GDP.pdf evaluates 100 practitioner-authored questions over original PDFs in ten domains, with expert rubrics, capability tags, and items screened for frontier-model failures.
Source: §1 · pp. 1–2, §3.1 · p. 2Results
The reported pass-rate range was 2–30.7%, and no frontier model passed even a third of the items.
Source: §5.2 · p. 5, §6 · p. 7Takeaways & Limitations
Failures cluster around tables, charts, footnotes, exclusions, noisy scans, and amendments, and the reported range does not support unsupervised use.
Source: §7 · p. 7, §6 · p. 7Because items were selected after defeating at least two frontier models, absolute pass rates measure adversarially selected tasks, not professional document work at large.
Source: §6 · p. 7
Abstract
from arXiv · showhide
A large share of day-to-day work in professional domains happens inside PDF files: benefits packets, leases, datasheets, clinical guidelines, construction plans. Benchmarks for document AI have generally measured the required capabilities in isolation: OCR, layout analysis, chart reasoning, table QA, document VQA. A high score on any one of them does not necessarily reveal whether a model can answer a realistic question that someone in the field would actually ask about a specific PDF. GDP_pdf is a benchmark built to measure this directly. It consists of question-document pairs authored by working professionals in ten fields, and a candidate question was kept only when at least two frontier multimodal models failed it in a way that mattered: a wrong answer, missed decisive evidence, or a fabricated claim, rather than a superficial difference such as style. Each item comes with a rubric of atomic criteria, so we can report a graded rubric score as well as a strict task-level pass rate, and each item is tagged against a taxonomy of eleven capabilities in three tiers, spanning text extraction and grounding, table and chart comprehension, cross-referencing, spatial reasoning, and abstention on unsupported queries. We report results for seventeen frontier models on the 100-item benchmark: the best model passes only 30.7% of the items and the worst passes 2%. Most errors trace back to a small set of recurring loss patterns: misaligned tables, misread charts, skipped footnotes and exclusions, miscounted floor-plan symbols, scan noise, and amendments that supersede earlier text.
1. Introduction
Professional PDF questions require more than isolated visual QA: decisive evidence may be distant, embedded in document structure, or require domain knowledge. GDP.pdf jointly evaluates these challenges using practitioner-authored tasks, expert rubrics, and capability tags.
1. Introduction
- GDP.pdf targets document structure, background knowledge, and failures where a real clause or number is cited but does not govern the question.Standard visual-reasoning scores are described as a poor guide to professional document workflows.Source: §1 · p. 1
- The benchmark contains 100 practitioner-authored items across ten domains, using original PDFs and evidence-based grading; each item had to defeat at least two frontier models.The full benchmark is released with source PDFs, prompts, rubrics, and domain labels.Source: §1 · p. 1
- GDP.pdf contributes an eleven-axis, three-tier capability taxonomy, atomic-rubric evaluation with graded scores and strict pass rates, and analysis of seventeen frontier models.The taxonomy organizes tasks and error analysis across foundational extraction and grounding, structural and multimodal comprehension, and advanced reasoning.Source: §1 · p. 2
2. Related Work
Prior work measures document reading, document question answering, specialized chart and table reasoning, or broad multimodal performance, often in settings that simplify the source. GDP.pdf centers professional questions whose answers depend on document constructs and evidence across the original file.
2. Related Work
- Parsing, OCR, and layout benchmarks measure page reading rather than whether models use what they read to complete paid tasks [12] [14] [9] [23] [36] [4].These benchmarks cover forms, receipts, page layout, and business-document extraction.Source: §2 · p. 2
- Document QA, long-document, and retrieval-aware work extends toward question answering and evidence finding, while GDP.pdf tests professional constructs such as footnotes and exclusions [20] [40] [16] [7] [27].Its questions may also hinge on legends or superseded sections.Source: §2 · p. 2
- Chart and table benchmarks commonly use cleaned inputs such as extracted text, an isolated table, or a chart, removing some difficulties of real PDFs [18] [22] [19] [3] [5].The passage also names finance, legal, and clinical domain-specific benchmarks.Source: §2 · p. 2
- Broad multimodal and knowledge-focused VQA suites serve different purposes and were not designed to show whether models can be trusted with a benefits packet or deed [15] [2] [17] [28].Table 1 positions GDP.pdf relative to these benchmark families.Source: §2 · p. 2
3. GDP.pdf: Benchmark Design
GDP.pdf frames each benchmark item as a professional PDF question paired with an expert rubric and evaluates both complete solutions and partial progress. Its design combines expert-authored, model-challenging tasks with domain coverage, a capability taxonomy, and a curation workflow that surfaced confident errors on short questions and superseded text.
3.1. Task Formulation
- Models are ranked by strict pass rate, while mean rubric scores preserve partial credit and expose which criteria are satisfied.Each item is represented by a PDF, question, expert rubric, reference answer, domain, and capability tags; a strict pass requires every rubric element to be satisfied.Source: §3.1 · p. 2
3.2. Design Principles
- Tasks reflect real professional workflows, require evidence from the PDF, and are retained only when at least two frontier models make a major error.Candidates are also filtered for ambiguity and reliance on general prior knowledge; rubrics specify required facts, acceptable equivalents, and claims to avoid.Source: §3.2 · p. 3, §3.5 · p. 4
3.3. Benchmark Profile and Coverage
- The benchmark spans financial filings, benefits packets, datasheets, clinical guidelines, plans, policies, deeds, inspection reports, and scanned amendments; answerable-without-document tasks are rejected.The current version's profile and domain coverage are summarized in Tables 2 and 3.Source: §3.3 · p. 3, Table 2 · p. 3, Table 3 · p. 4
3.4. Capability Taxonomy
- Eleven capability axes are organized into three tiers, and most items receive multiple tags used to guide task authoring and error analysis.The tiers cover extraction and grounding, structural and multimodal comprehension, and advanced reasoning.Source: §3.4 · p. 3, Table 4 · p. 4
3.5. Collection and Curation Workflow
- Experts source work tasks, calibrate difficulty with frontier models, filter for novelty and ambiguity, author atomic rubrics, tag capabilities, and run sanity checks.At least two screening models must make a major failure, such as a materially wrong answer, omitted decisive evidence, or fabricated claim.Source: §3.5 · pp. 3–4
3.6. Data Curation Insights
- Short questions about a single table or figure can be among the hardest, while cross-section comparisons and amendments overriding earlier text often prompt confident hallucinations.Table 5 gives sample items, deciding evidence, and common model errors.Source: §3.6 · pp. 4–5
4. Evaluation Protocol
The evaluation scores responses against atomic criteria rather than relying on textual overlap, including when the document does not support an answer. Rubric scores support slice analysis and strict pass rates rank models; an LLM judge grades criteria separately, with failures manually checked.
Atomic rubric design
Unsupported queries and abstention
- Unsupported-query tasks require explicit abstention when evidence is absent, and fluent fabrication receives zero credit.Examples include redacted figures, unreported values, and conclusions that do not follow from the document.Source: §4 · p. 5
Slice reporting
- Rubric scores retain partial progress for slices, while strict pass rate credits only fully solved items for model ranking.Failure analysis draws qualitatively on slice results.Source: §4 · p. 5
Judging and manual verification
- An LLM judge grades each self-contained criterion without seeing the source PDF or gold answer; reported failures are also checked by hand against the document and collector notes.The judge is calibrated against expert raters, and its model family is among those evaluated.Source: §4 · p. 5
5. Evaluation
The evaluation tests public-API models on original PDFs without tools or added context, then analyzes their strict-pass results and recurring document-reading failures. No evaluated model passes a third of the items, with errors spanning tables, plots, cross-references, amendments, prior knowledge, and spatial tasks.
5.1. Evaluated Models
- Models receive the question and original PDF through public APIs, with no tools or additional context; reported strict pass rates average five runs.Provider document handling is part of what is measured.Source: §5.1 · p. 5
5.3. Failure Analysis
- Failures often turn on small decisive cues: models misalign merged-cell or stacked-header tables and can return implausible values from the correct plot.The table examples include selecting the wrong row or reporting numbers absent from the selected band.Source: §5.3 · p. 5, Table 5 · p. 6
5.2. Main Results
- No evaluated model passes a third of the items; the best fails more than two-thirds, and three models pass just 2%.Table 6 reports strict pass rate averaged over five runs per model.Source: §5.2 · p. 5, Table 6 · p. 6
5.3. Failure Analysis
- Models often miss controlling footnotes, cross-references, and exclusions, including a footnote that changes a leave ranking and exclusions that determine insurance coverage.In the insurance example, models stopped at the insuring agreement before reaching the exclusions.Source: §5.3 · pp. 5–6
- When document guidance conflicts with prior knowledge, models can give plausible but document-wrong advice, as with the specified drill-bit recommendation.The document advised against brad-point bits for the application described.Source: §5.3 · p. 6
- Spatial tasks require matching small symbols to legends and linking pages; models misplaced fixtures and miscounted windows when they failed to connect a schedule to a plan.These spatial slices were among the hardest task types.Source: §5.2 · p. 5, §5.3 · p. 6
- Long noisy files and amendments produced cascading errors, including models quoting deed sections that an amendment had already replaced.In an automated valuation report, every model in the initial evaluation round misread a “PENDING” label’s relevance.Source: §5.3 · p. 6
6. Discussion
Discussion results show a wide gap between broad multimodal leaderboard standing and performance on adversarially selected professional-document tasks. The authors connect this gap to evidence grounding, discuss the benchmark’s reliability implications, and identify development directions.
6. Discussion
- 30.7% and 29.8%: the best two models pass only these shares of GDP.pdf items despite ranking at or near the top of most public multimodal leaderboards.Because each item defeated at least two frontier models during collection, absolute pass rates describe adversarially selected tasks, not professional document work at large.Source: §6 · p. 7
- Many failures mix perception and reasoning: models read an insurance agreement correctly but granted coverage without checking the relevant exclusion.The authors note that a policy reader would know to check for the exclusion.Source: §6 · p. 7
- The authors frame GDP.pdf as a reliability “floor”: its 2–30.7% pass-rate range does not support unsupervised document automation.They contrast this with broader benchmarks’ “ceiling” question about tasks models can accomplish in controlled academic settings.Source: §6 · p. 7
- The benchmark draws on routine work contributors actually perform, including insurance exclusions, HR leave announcements, and estimator window counts.The authors say broader benchmarks sample these skills thinly.Source: §6 · p. 7
- The authors suggest preserving page structure, improving chart and table parsing, retaining visual evidence in retrieval, treating footnotes and amendments as content, and improving uncertainty calibration.They state longer context windows alone might not fix the capability gap.Source: §6 · p. 7
7. Conclusion
GDP.pdf benchmarks multimodal reasoning grounded in professional PDFs using expert-authored tasks and original files with domain semantics intact. Across seventeen frontier models evaluated as of July 2026, none passes a third of the items on the strict metric; failures concentrate in difficult document elements.
7. Conclusion
- Across seventeen frontier models evaluated as of July 2026, none passes a third of GDP.pdf items on the strict metric.Failures concentrated in tables, charts, footnotes, exclusions, noisy scans, and amendments that supersede earlier text.Source: §7 · p. 7