PaperlayerView exact revision on arXiv

Source-linked AI summary

LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating

Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, Jian Xu, Xiao-Hui Li, Yuan Gao, Jun Song, Bo Zheng, Cheng-Lin Liu

arXiv:2412.18424v3cs.AIcs.CL

TL;DR

Existing document benchmarks cover limited page lengths and do not comprehensively assess cross-element locating. LongDocURL organizes 20 subtasks across understanding, numerical reasoning, and locating, and evaluates models on 2,325 QA pairs from more than 33,000 pages; GPT-4o scores 64.5 versus 30.6 for the best open-source model.

  • Problem

    Existing benchmarks are limited in page coverage and provide insufficient assessment of complex document elements, diverse tasks, and cross-element locating.

    Source: §1 · p. 1

  • Method

    LongDocURL defines three task categories and 20 subtasks, then uses a semi-automated pipeline to build and verify document QA pairs.

    Source: §1 · p. 2

  • Results

    GPT-4o scored 64.5, leading all models, while the best open-source model scored 30.6 across the evaluation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that broader data sources could enrich document-layout evaluation, and that the dataset can be expanded through automated construction.

    Source: Limitations · p. 9

    The document-source types covered in the study remain limited.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large vision language models (LVLMs) have improved the document understanding capabilities remarkably, enabling the handling of complex document elements, longer contexts, and a wider range of tasks. However, existing document understanding benchmarks have been limited to handling only a small number of pages and fail to provide a comprehensive analysis of layout elements locating. In this paper, we first define three primary task categories: Long Document Understanding, numerical Reasoning, and cross-element Locating, and then propose a comprehensive benchmark, LongDocURL, integrating above three primary tasks and comprising 20 sub-tasks categorized based on different primary tasks and answer evidences. Furthermore, we develop a semi-automated construction pipeline and collect 2,325 high-quality question-answering pairs, covering more than 33,000 pages of documents, significantly outperforming existing benchmarks. Subsequently, we conduct comprehensive evaluation experiments on both open-source and closed-source models across 26 different configurations, revealing critical performance gaps in this field.

1 Introduction

The introduction argues that existing benchmarks do not adequately cover long documents, complex elements, and cross-element tasks. It presents LongDocURL as a 20-subtask benchmark and reports evaluation gaps between current models.

1 Introduction

  • Existing benchmarks have limited page coverage, sparse cross-element assessment, and insufficient coverage of complex elements and diverse tasks.The introduction notes that some multipage benchmarks do not exceed 20 pages, while MMLongBench-Doc has about 1k effective samples and about 30% cross-page questions.Source: §1 · pp. 1–2
  • LongDocURL defines Understanding, Numerical Reasoning, and Cross-Element Locating, including tasks that require switching between element types.Para-Title Locating requires matching a given abstract to relevant paragraphs and identifying their section titles.Source: §1 · pp. 2–3
  • GPT-4o scored 64.5, leading all models, while the best open-source score was 30.6, indicating a potential document-understanding gap.The benchmark contains 2,325 QA pairs covering more than 33,000 pages and evaluates 26 configurations.Source: §1 · p. 2
  • The benchmark uses a semi-automated pipeline to generate and verify QA pairs from documents, then evaluates open- and closed-source models across 26 configurations.Its construction pipeline includes document extraction and filtering, QA generation, automated verification, and human verification.Source: §1 · pp. 2–3

2 Related Work

Related work distinguishes OCR-dependent and end-to-end document models, reviews approaches to long-document processing, and surveys benchmarks for multipage understanding and retrieval.

Models for Document Understanding

  • OCR-dependent models extract text for processing, whereas end-to-end models combine visual-encoder features with language-model backbones.Examples include LayoutLM and text-only LLMs for the former, and GPT4o, Gemini-1.5, and Qwen2-VL for the latter.Source: §2 · p. 3

Methods for Long Document Understanding

Benchmarks for Long Document Understanding

  • Existing benchmark coverage ranges from multipage QA to long-document understanding and retrieval, while MP-DocVQA and DUDE do not assess documents exceeding 20 pages (Ma et al., 2024; Chia et al., 2024; Ding et al., 2024a,b; Wang et al., 2024).MMLongBench-Doc and M-Longdoc average 47.5 and 210.8 pages per document, respectively.Source: §2 · p. 3
  • The paper’s construction pipeline is organized into Extract & Filter, QA Generation, Automated Verification, and Human Verification modules.Figure 3 presents the pipeline overview.Source: Figure 3 · p. 4

3 LongDocURL

LongDocURL classifies QA pairs by task, evidence-page count, and evidence-element types, then builds and verifies a large document QA dataset. The resulting 396-document collection contains 2,325 samples, including a substantial cross-element locating portion.

3.1 Overview

  • LongDocURL organizes 20 subtasks across Understanding, Reasoning, and Locating, with evidence classified by page count and element type.Understanding and Reasoning have eight subtasks based on evidence-page count; Locating has four based on combinations of element types.Source: §3.1 · p. 4, Figure 4 · p. 5

3.2.1 Evidence Collection

  • From 200,000 CommonCrawl PDFs, the authors filtered English documents of 50–150 pages and retained 396 documents across eight types, averaging 85.6 pages.They used PyMuPDF and Docmind to extract region-level text-type-bbox triples, including paragraphs, tables, charts, footnotes, and titles.Source: §3.2.1 · p. 4

3.2.2 Q&A Generation

  • QA generation uses parsed text-type-bbox triples and multi-step LLM/LVLM prompting because direct generation from document images was often ineffective.Prompts include task definitions and constraints.Source: §3.2.2 · p. 5

3.3.1 Automated Verification

  • Automated verification checks task relevance, format correctness, and faithfulness, but marked about 75.2% of raw Cross-Title Locating samples and 19.6% of Cross-Table Locating samples negative.Verification outcomes and their text chains are passed to human annotators for further review.Source: §3.3.1 · p. 5

3.3.2 Human Verification

  • Human verification rechecks machine decisions, checks evidence against the source visuals, and cross-checks annotations between annotators.The human stage addresses automated verification’s text-only comparison and its failure to recycle negative samples.Source: §3.3.2 · pp. 5–6

3.4 Dataset Statistics

  • The dataset contains 396 documents and 2,325 QA samples, with documents averaging 50–150 pages and cross-element locating contributing 37.1% of samples.Five answer types support automated evaluation while maintaining completeness and accuracy.Source: §3.4 · p. 6

4 Experiments

The experiments compare models and input formats across 26 configurations, reporting a large gap between top closed- and open-source LVLMs. Fine-grained analyses and input-paradigm comparisons expose variation by task, document element, and parsing method.

4.1 Evaluation Protocols

  • The evaluation uses response generation, answer extraction, and score calculation, with five answer categories and a softer scoring standard than MMLongBench-Doc.The answer extractor converts lengthy responses into concise answers for rule-based scoring.Source: §4.1 · p. 6

4.2 Experimental Setup

  • The experiments use text-input and image-input configurations, with documents up to 150 pages evaluated using a cut-off paradigm because many models cannot process the full context.LVLMs receive images of 30 continuous pages around the answer evidence; LLMs receive OCR-parsed text.Source: §4.2 · p. 6

4.3 Main Results

  • GPT-4o scored 64.5, the highest LVLM score, while Qwen2-VL led open-source models at 30.6; only GPT-4o met the passing standard.The table reports generalized accuracy across task types, evidence elements, and evidence-page or cross-element categories.Source: §4.3 · pp. 6–7, Table 3 · p. 7
  • Top LLM scores trailed top LVLM scores by about 30 points, with plain-text parsing associated with loss of document structure, especially in tables and charts.The paper identifies this benchmark as a test of LVLM document-structure parsing capability.Source: §4.3 · p. 7

4.3.1 Fine-Grained Analysis

  • Models scored highest on text and lowest on table questions; weaker models also performed poorly on locating, while GPT-4o reasoning scores fell 31.6 points with text input versus 22.4 for locating.Single-page QA accuracy was lower than multi-page accuracy overall, although GPT-4o and Qwen-VL-Max showed lower accuracy on multi-page QA.Source: §4.3.1 · pp. 7–8, Figure 5 · p. 8

4.4 Ablation of Input Paradigms

  • Cut-off input scored above PyMuPDF text input but below Docmind text input, and cut-off may outperform merging; Docmind’s advantage over PyMuPDF was at least 25 points for proprietary models and 15 for open-source models.The authors attribute the parsing difference to Docmind retaining more table structure than PyMuPDF; one Qwen2-VL experiment was not completed due to resource limitations.Source: Table 4 · p. 8, §4.4 · p. 8

5 Error Analysis and Model Insights

The error analysis identifies perceptual mistakes as the most common GPT-4o errors, while model comparisons highlight multimodal training and preference tuning. Layout analysis and table/chart parsing remain frequent failure areas.

Error Analysis

  • 32.7% of sampled GPT-4o errors were perceptual, involving inaccurate recognition, parsing, or counting of document elements.Complex document structures and OCR performance are identified as primary bottlenecks for LVLMs.Source: §5 · p. 8
  • 20.6% of errors involved inconsistent formats, while reasoning errors were the third-largest category at 16.8%.Hallucinated evidence, irrelevant answers, incomplete evidence, and extractor errors collectively accounted for 29.9% of cases.Source: §5 · p. 9

Model Insights

  • Qwen2-VL outperforms its LLM variant, and LLaVA-OneVision-Chat outperforms its base model, with multimodal training and DPO cited as explanations.The analysis also notes frequent failures in layout analysis and table/chart parsing; it suggests layout-aware training or parsing before reasoning as possible mitigations.Source: §5 · p. 9

6 Conclusion

LongDocURL addresses limitations in document benchmarks with 20 capabilities spanning three tasks and three evidence modes. Its construction and model evaluations cover more than 33,000 document pages and reveal potential gaps in document understanding.

6 Conclusion

  • LongDocURL contains 20 capabilities across 3 tasks, 3 evidence modes, and 4 document elements, built from over 2,300 QA pairs spanning more than 33,000 pages.The study evaluates 26 configurations of open- and closed-source models and reports potential gaps in document understanding.Source: §6 · p. 9

Limitations

The paper identifies limited document-source variety as a dataset boundary and describes the benchmark’s distribution across several dimensions.

Limitations

  • The document sources remain limited; broader sources could provide richer layouts and element information, while pipeline automation could further expand the dataset.Improving model structures and training processes is identified as important but beyond the paper’s scope.Source: Limitations · p. 9

A More Statistics of LongDocURL

  • Figure 8 characterizes LongDocURL by document-page counts, answer-evidence pages, evidence-page length, document sources, and evidence-element types.The supplied text identifies these distribution dimensions but gives no values.Source: A More Statistics of LongDocURL · p. 12

B Method Details

The appendices specify how QA pairs are generated and checked, how document images are selected and prompted, and how answers are extracted and scored. They also document verification statistics and dataset or input-paradigm comparisons.

B.1.1 Prompt Template for QA Generation

  • QA generation uses structured text and possible intermediate responses, and requests definite, image-answerable questions with evidence descriptions before short answers.Intermediate responses can include summaries of paragraphs under section titles for Para-Title Locating data construction.Source: §B.1.1 · p. 12

B.1.2 Prompt Template for QA Verification

  • QA verification checks task fit, formatting, relevance, clarity, precision, and coherence against the document context.The process outputs yes when the criteria are met and no otherwise.Source: §B.1.2 · p. 12

B.1.3 Statistics of QA Verification

  • Table 5 is described as reporting changes in the amount of data before and after QA verification.The supplied passage gives the table’s scope, not its values.Source: Table 5 · p. 13

C Comparison with MMLongBench-Doc

  • Figure 9 compares LongDocURL and MMLongBench-Doc by dataset attributes.The supplied caption does not state the comparison’s outcomes.Source: Figure 9 · p. 13

D.1 Merging Rules of Images Input

  • The image-input merge paradigm is referenced as an evaluation rule; Table 6 describes ablation rules by merged-array columns and resulting image counts.The supplied passages do not specify the individual merge configurations.Source: §D.1 · p. 13, Table 6 · p. 13

D.2 Selection Rules of Images Input

  • The cutoff paradigm selects input pages around the QA context, using a 30-image input limit and adjusting the range at document boundaries.The pseudo-code returns the selected start and end page indices.Source: §D.2 · pp. 13–14

D.3 Prompt for Response Generation

  • The response-generation prompt asks models to answer questions using the provided document images.The prompt includes the question as input.Source: §D.3 · p. 14

D.4 Prompt for Answer Extraction

  • The answer-extraction prompt modifies the MMLongBench-Doc template, with changes marked in blue in the referenced figure.The supplied passage does not describe the individual modifications.Source: §D.4 · p. 14

D.6 Fine-Grained Evaluation Results

The fine-grained results are presented in Table 7, with related analysis in Section 4.3.1. The case study shows responses from different models and provides a short error analysis for each case.

D.6 Fine-Grained Evaluation Results

  • Detailed fine-grained results are presented in Table 7, with related analysis in Section 4.3.1.The supplied passages do not report the table’s individual scores.Source: §D.6 · p. 14

E Case Study

  • Figures 11 and 12 display different models’ responses, each accompanied by a short error analysis.Source: E Case Study · p. 14

F Data Examples

The examples cover LongDocURL’s three primary tasks and include an evaluation answer-extraction prompt and model case studies. Table 7 defines the task, evidence-element, and evidence-page categories used for its generalized accuracy scores.

F Data Examples

  • Figures 13–15 provide sample questions for the understanding, reasoning, and locating tasks.Source: F Data Examples · p. 14, Figure 13 · p. 19, Figure 14 · p. 20
  • Figure 10 shows the prompt used for answer extraction during evaluation.Source: Figure 10 · p. 15
  • Table 7 reports generalized accuracy scores across understanding, reasoning, and localization, with evidence categorized by element type and page or cross-element scope.Its categories include text, layout, chart and image, and table evidence; single-page, multi-page, and cross-element evidence; the highest scores in each section are highlighted in green.Source: Table 7 · p. 16
  • Figures 11 and 12 show model responses for table evidence on pages 26–27 and 110–111, with correct and incorrect extractions and reasoning color-coded.Source: Figure 11 · p. 17, Figure 12 · p. 18

G Error Analysis

The authors examine a random sample of 97 GPT-4o errors and describe several recurring error patterns. They also report separate annotator groups for human verification and final quality checks.

G Error Analysis

  • The error analysis examines 97 randomly sampled errors from GPT-4o, identified as the best-performing model.Source: G Error Analysis · p. 22
  • Perceptual errors involve failures to recognize, parse, or enumerate visual elements, while reasoning errors occur during calculations, comparisons, or summarization despite correct evidence retrieval.Source: G Error Analysis · p. 22
  • Formatting discrepancies include abbreviation/full-form differences, omitted or added numerical units, and decimal-precision mismatches with ground truth.Source: G Error Analysis · p. 22
  • Incomplete-evidence errors omit query-relevant information; irrelevant answers misinterpret query intent; hallucinated evidence fabricates document content.Source: G Error Analysis · p. 22

H Description of labeling labor

  • Twenty-one full-time annotators performed human verification, while six postgraduate-level professional annotators handled final quality verification and cross-checking.Source: H Description of labeling labor · p. 22
Loading 2412.18424v3…