DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems

1 Introduction

• Document-reading systems must parse documents and layouts, extract metadata, handle embedded images, and manage long textual contexts.These demands make document queries more complex than standalone LLM text interactions.Source: §1 · pp. 1–2
• The absence of a standardized benchmark makes objective, fine-grained evaluation and comparison of document-reading systems difficult.The paper presents DocBench as a benchmark for real-world inputs consisting of a document and related questions.Source: §1 · p. 2
• DocBench contains 229 real-world documents and 1,102 questions across five domains and four question categories.Questions cover text-only, multimodal, metadata, and unanswerable cases.Source: §1 · p. 2
• DocBench combines human annotation and synthetic question generation to support fair comparisons across document-reading systems.The benchmark is intended to highlight current limitations and guide future advancements.Source: §1 · pp. 2–3
• Evaluations of proprietary systems and an open-source parse-then-read pipeline reveal noticeable gaps between existing systems and human performance.Proprietary systems are accessible via web interfaces or APIs, but their operational strategies are not fully disclosed.Source: §1 · p. 2

2.1.1 Document Collection

• Documents are gathered from five domains, yielding 229 PDFs: 49 academia, 40 finance, 44 government, 46 laws, and 50 news.Sources include arXiv, corporate annual reports, U.S. government reports, Library of Congress publications, and scanned New York Times front pages.Source: §2.1.1 · p. 3

2.1.2 QA-pair Generation

• QA generation uses parsed PDF text for GPT-4 text-only questions, GPT-4V for multimodal questions, and annotators for 350 additional pairs.Annotators focus on metadata and unanswerable questions, which were less covered in the preceding generation stage.Source: §2.1.2 · p. 4

2.1.3 Quality Check

• GPT-4 filters questions before manual review, and seven domain practitioners review and refine data in their areas of expertise.Source: §2.1.3 · p. 4

2.2 Dataset Statistics

• 229 PDF documents and 1,102 questions span five domains; each document has 1–16 QA pairs, within a stated maximum of 20.Question and document token counts range from 6 to 40 and from 1,300 to 598,302, respectively.Source: §2.2 · p. 4

2.3 Dataset Analysis

• Text-only questions make up 37.4% of QA pairs, multimodal 27.9%, metadata 23.4%, and unanswerable 11.3%.Source: §2.3 · p. 5
• Questions are categorized as what/who/where/when/which (58.6%), yes/no (22.1%), how (18.8%), and why (0.5%).Source: §2.3 · p. 5
• Answers are numerical (37.4%), textual (35.7%), boolean (17.3%), or other (9.6%); 69% of numerical answers come from academia and finance.Source: §2.3 · p. 5

2.4 Evaluation Setup

• GPT-4’s automatic evaluator agrees with human annotators on 98% of evaluated examples.The evaluation uses binary accuracy scores and reports results overall and by domain and QA-pair type.Source: §2.4 · p. 6, Table 3 · p. 6

3.1 Experimental Setup

• 22 systems were evaluated, including proprietary document-upload systems and parse-then-read pipelines using fitz to extract PDF text and image blocks.The open-source base models ranged from 7B to 70B and were selected for instruct/chat availability, vLLM support, and context length/version.Source: §3.1 · p. 6

3.2 Results and Discussion

• DOCBENCH results vary substantially across systems, with differences associated with foundational models, context limits, and design and implementation approaches.Table 4 reports results across question types and domains.Source: §3.2 · p. 7, Table 4 · p. 7

3.2.1 Interpreting Multi-modal and Metadata Information

• Multimodal questions can fail at locating the relevant figure, extracting its data, or calculating from the extracted values.In the case study, KimiChat missed the chart, Claude-3 reported 288 and 348 instead of 326 and 390, and GPT-4 miscalculated the percentage change despite locating and extracting correctly.Source: §3.2.1 · p. 7, Figure 5 · p. 8
• Metadata-question performance is generally below 55%, which the authors attribute to limited attention to global information.Source: §3.2.1 · p. 8

3.2.2 Handling Lengthy Documents

• Both methods perform poorly on finance documents but better on news, while 8k Llama-3 models that excel on news are outperformed by all 128k models on finance.The authors link the domain difference to finance files being longer and richer in information than single-front-page news files.Source: §3.2.2 · pp. 8–9, Figure 6 · p. 8

3.2.3 Faithfulness to User-provided Documents

• Most systems struggle with unanswerable questions; GPT-4 scores 37.1% as a document-reading system versus 70.2% in the parse-then-read pipeline.Gemma and KimiChat perform better in these scenarios; the authors identify fidelity to the supplied documents as a key challenge.Source: §3.2.3 · p. 9

4.1 Recent Advances of LLMs and LLM-based Systems

• LLM-based systems combine language models with architectures and components to handle tasks involving external resources and multiple reasoning, execution, and reflection steps.Source: §4.1 · p. 9

4.2 Document reading: Datasets and Methods

• Existing text-focused document and long-context datasets omit multimodal document structure, while multimodal datasets often cover only parts such as tables or figures.DOCBENCH instead evaluates systems on full documents as intact files and varied question types.Source: §4.2 · pp. 9–10

5 Conclusion

• DOCBENCH contains 229 documents and 1,102 questions across 5 domains and 4 question types, developed with human annotators and synthetic questions.Source: §5 · p. 10
• The evaluation covers proprietary LLM systems and open-source parse-then-read approaches, and finds significant disparities in document-reading capabilities.Source: §5 · p. 10

A.1 Response Evaluation

• Response-evaluation instruction prompts are presented in Table 5.Source: §A.1 · p. 13, Table 5 · p. 14

A.2 QA-pair Generation

• Passing diagrams to GPT-4V alone produces subpar question quality.Source: §A.2 · p. 13
• The authors suggest diagrams without accompanying text descriptions may lack enough information, causing generated QA pairs to deviate from their intended meanings.Source: §A.2 · p. 13
• Adding Easy, Medium, and Hard difficulty settings to QA-generation prompts can result in higher-quality pairs.Source: §A.2 · p. 13
• The authors suggest difficulty settings may let the model favor higher generation quality in potential comparisons.Source: §A.2 · p. 13
• Instruction prompts for generating QA pairs are provided in Table 6.Source: §A.2 · p. 13

• GPT-4 performs consistently high and balanced across aspects in the parse-then-read pipeline.Source: B Performance Comparison · p. 14
• KimiChat scores consistently high across metrics among LLM-based systems, demonstrating balanced performance.Source: B Performance Comparison · p. 14
• GPT-4 performs poorly on unanswerable questions among LLM-based systems, while Claude-3 excels on metadata questions.The passage associates GPT-4's result with potential overfitting in optimized GPT-4 file systems and decreased fidelity to documents.Source: B Performance Comparison · p. 14
• For parse-then-read pipelines, GPT-4 and Gemma differ clearly from the remaining methods on multimodal and unanswerable questions.Source: B Performance Comparison · p. 14

C Analysis of Input Sources

• Without attached files, GPT-4 correctly answers 19.1% of questions and GPT-4o correctly answers 21.7%.Source: C Analysis of Input Sources · p. 14, Table 7 · p. 16
• Correct answers without files are predominantly textual and largely associated with government, law, and news domains.Source: C Analysis of Input Sources · pp. 14–15
• The authors suggest the models' training data is heavily skewed toward these categories, enabling accurate answers to some questions without additional files.They also suggest GPT-4o may benefit from broader and more extensive training data than GPT-4.Source: C Analysis of Input Sources · p. 15