PaperlayerView exact revision on arXiv

Source-linked AI summary

DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models

Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wenjie Wu, Hancheng Ye, Shiyang Feng, Bin Wang, Chao Xu, Conghui He, Pinlong Cai, Min Dou, Botian Shi, Sheng Zhou, Yongwei Wang, Bin Wang, Junchi Yan, Fei Wu, Yu Qiao

arXiv:2406.11633v2cs.CV

TL;DR

Scientific-document understanding remains challenging for MLLMs because of complex modalities and logical relationships. DocGenome structures 500K arXiv documents into a multi-task benchmark, and experiments report that it can enhance existing baselines’ document understanding.

  • Problem

    Scientific documents contain complex charts, equations, and logical relationships that MLLMs still struggle to parse and comprehend.

    Source: §1 · p. 1

  • Method

    DocParser automatically structures 500K arXiv documents from 153 disciplines into a multi-modal benchmark with component attributes, relationships, and seven document-oriented tasks.

    Source: §1 · p. 2

  • Results

    Experiments on DocGenome report that the benchmark can enhance document understanding in existing baseline models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DocGenome provides a structured resource for training scientific-document models and evaluating large models’ capabilities on document tasks.

    Source: §6 · p. 9

    The impact of entity-relationship annotations on large models’ scientific-document understanding remains to be explored.

    Source: §B.1 · p. 14

Abstract

from arXiv · show

Scientific documents record research findings and valuable human knowledge, comprising a vast corpus of high-quality data. Leveraging multi-modality data extracted from these documents and assessing large models' abilities to handle scientific document-oriented tasks is therefore meaningful. Despite promising advancements, large models still perform poorly on multi-page scientific document extraction and understanding tasks, and their capacity to process within-document data formats such as charts and equations remains under-explored. To address these issues, we present DocGenome, a structured document benchmark constructed by annotating 500K scientific documents from 153 disciplines in the arXiv open-access community, using our custom auto-labeling pipeline. DocGenome features four key characteristics: 1) Completeness: It is the first dataset to structure data from all modalities including 13 layout attributes along with their LaTeX source codes. 2) Logicality: It provides 6 logical relationships between different entities within each scientific document. 3) Diversity: It covers various document-oriented tasks, including document classification, visual grounding, document layout detection, document transformation, open-ended single-page QA and multi-page QA. 4) Correctness: It undergoes rigorous quality control checks conducted by a specialized team. We conduct extensive experiments to demonstrate the advantages of DocGenome and objectively evaluate the performance of large models on our benchmark.

1 Introduction

DocGenome addresses the difficulty MLLMs have parsing scientific documents’ complex modalities and logical relationships. It introduces a large-scale, structured benchmark and reports that it can enhance existing baselines’ document understanding.

1 Introduction

  • Scientific documents offer high-quality, logically rich training data, but MLLMs struggle with their complex charts, equations, and logical relationships.Source: §1 · p. 1
  • DocParser automatically annotates and structures arXiv papers to produce component-unit attributes and relationships, with quality grading and human validation supporting the dataset.Source: §1 · p. 2
  • The benchmark covers seven document-oriented tasks, including classification, visual grounding, layout detection, document transformation, and single- and multi-page question answering.Source: §1 · p. 2
  • Experiments on DocGenome report that the benchmark can enhance document understanding in existing baseline models.Source: §1 · p. 2
  • DocGenome structures 500K scientific documents with 13 component-unit categories and six relationship types, spanning eight primary and 153 secondary disciplines.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Works

Earlier visual document datasets emphasized recognizing document regions, while later work broadened dataset diversity. DocGenome is compared with document-related benchmarks across several dimensions.

2 Related Works

  • Earlier visual document datasets focused mainly on recognizing document-region categories such as text, tables, and abstracts [22] [56] [9].Source: §2 · p. 2
  • Later datasets expanded diversity across tasks, modalities, and training data [32] [51] [52] [40] [21] [25].Source: §2 · p. 3
  • DocGenome is described as more comprehensive in disciplines, training samples, task types, evaluation metrics, and entity relationships than compared benchmarks.Table 1 presents the comparison with document-related benchmarks.Source: §2 · p. 3, Table 1 · p. 3
  • Visual document AI has progressed in layout analysis [44] [40] [9] [3] [30] [17] [14] and image representation learning [57] [13] [10] [6], alongside unified vision tasks [37], OCR-free understanding [54], and information extraction [12].Source: §2 · p. 3
  • The related work also surveys commercial and open-source MLLMs, including GPT-4V and the Gemini series [38] [35].Source: §2 · p. 3

3 Data Collection Methodology For DocGenome

DocParser structures arXiv source documents into component-level annotations, relationships, and bounding boxes. DocGenome’s analysis reports broad disciplinary and temporal coverage, annotation volumes, and quality tiers.

3.1 Introduction of Auto-labeling Pipeline

  • DocParser applies four stages—preprocessing, unit segmentation, attribute assignment and relation retrieval, then color rendering—to produce unit annotations and a rendered document.The annotations include component source code, attributes, relationships, and bounding boxes.Source: §3.1 · pp. 3, 5, Figure 2 · p. 4
  • Preprocessing prepares LaTeX for compilation and rendering; segmentation preserves reading order, while attribute assignment classifies units and retrieves relationships.For bounding boxes, DocParser renders the target unit black and other units white, then subtracts the PDFs; units spanning pages receive unified labels.Source: §3.1 · pp. 3–5

3.2 DocGenome Benchmark Analyses

  • The corpus spans 2007–2022, has a median publication year of 2016, and includes approximately 32.88% of articles published since 2020.It encompasses 8 primary disciplines and 153 secondary disciplines.Source: §3.2 · p. 5
  • The arXiv corpus comprises 500K academic articles; the annotated collection totals 74.5M component units and 68.5M relationship labels, with documents averaging 13 pages and the longest reaching 50 pages.Source: §3.2 · p. 5
  • The quality analysis uses IoUintra to measure overlap among a paper’s annotated boxes and IoUalign to compare annotations with DocXChain reference boxes.IoUintra assesses intra-consistency; IoUalign measures alignment with reference boxes predicted by DocXChain [53].Source: §3.2 · p. 5
  • Tier 1 contains 28.56% of papers, Tier 2 contains 61.30%, and Tier 3 contains 10.14%, under thresholds based on IoUintra and IoUalign.Tier 1 requires IoUintra < 0.05% and IoUalign > 60%; Tier 2 requires 0.05% ≤ IoUintra < 1% and IoUalign > 35%.Source: §3.2 · p. 5, Figure 3 · p. 6

4 DocGenome-test: A Multi-task, Multi-modal, Comprehensive Evaluation Set for Document Understanding

DocGenome-test is a discipline-balanced, precisely annotated evaluation set, with QA pairs designed to test both page-level and cross-page understanding. Its seven tasks span document classification, visual grounding, QA, layout detection, and document transformation.

4.1 Principles of Constructing Evaluation Set

  • The 1,004-paper DocGenome-test set is sampled uniformly across disciplines from Tier-1 papers and includes 9K pages plus examples for six task formats.It contains 1K classification examples, 2K visual-grounding examples, 3K QA pairs, 110K layout boxes, 3K Table-LATEX pairs, and 5K Equation-LATEX pairs.Source: §4.1 · p. 6

4.2 QA Pair Generation and Quality Assurance

  • Each paper receives two single-page and two cross-page QA pairs generated with GPT-4V, with cross-page pairs testing integration of information across pages.Faculty members review each pair: two independently check correctness, relevance, clarity, question rationality, and answer accuracy, then cross-check results against the source text.Source: §4.2 · p. 6

4.3 Evaluation Tasks

  • The seven tasks cover classification, visual grounding, single-page and multi-page QA, layout detection, and Equation-to-LATEX and Table-to-LATEX transformation.All tasks use paper images as visual input; grounding identifies content from visual components and text prompts, while layout detection localizes and recognizes blocks.Source: §4.3 · p. 7

4.4 Evaluation Metrics

  • Metrics pair Top-1 Accuracy and mAP@0.5:0.95 with higher-is-better scoring, while visual grounding uses lower-is-better Edit Distance.Transformation uses Edit Distance, Jaccard Similarity, Cosine Similarity, and BLEU; open-ended QA uses GPT-acc evaluated against ground truth with GPT-4.Source: §4.4 · p. 7

5 Experiments

Experiments show task-dependent MLLM performance on DocGenome-test and improvements from scaling DocGenome-train for layout detection and document transformation. The discussion also examines OOD generalization and potential uses of the dataset’s annotations and relationships.

5.1 Compared Baselines and Implementation

  • Baselines include general MLLMs and task-specific systems; inference uses document images, with multi-page QA requiring consecutive pages and multi-image model support.For transformation, layout annotations crop tables and equations before Pix2Struct-B-structure models are fine-tuned as EqVLM-B and TableVLM-B.Source: §5.1 · pp. 7–8

5.2 Performance on DocGenome-test

  • GPT-4V leads classification (98.0% Top-1 Acc), Title OCR Grounding (0.0104 Edit Distance), single-page QA (61.0% GPT-acc), and multi-page QA (65.0% GPT-acc); InternVL 1.5 leads Abstract OCR Grounding (0.3601 Edit Distance).The evaluation covers classification, visual grounding, and single-page and multi-page QA.Source: §5.2 · p. 8

5.3 Effectiveness of DocGenome-train

  • 48.43% to 86.52% mAP for Footnote detection accompanies scale-up from 7K to 700K training examples, and YOLOv8 outperforms DocXChain on seven shared attributes.YOLOv8 layout-detection capacity improves as training data increases.Source: §5.3 · p. 8, Table 4 · p. 8
  • Both transformation tasks benefit most from scaling training data from 10K to 100K; beyond 100K, TableVLM-B shows greater improvement than EqVLM-B.The comparison covers Table-to-LATEX and Equation-to-LATEX transformation.Source: Table 5 · p. 8, §5.3 · p. 9

5.4 Further Discussions

  • On OOD data, DocGenome-train YOLOv8 generalizes better than DocXChain on human-annotated layout data, while EqVLM-B records 0.6627 Edit Distance on Scihub equations.The authors note EqVLM-B performs relatively worse than Mathpix on OOD data.Source: §5.4 · p. 9, Table 6 · p. 9
  • The discussion proposes extending transformation to other modalities, using entity relations for document-level tasks, and applying annotations to OCR at any page location.The reported transformation experiments cover only Table-to-LATEX and Equation-to-LATEX.Source: §5.4 · p. 9

6 Conclusion

The conclusion presents DocGenome as a structured, multi-task, multi-modal scientific-document dataset built with DocParser to extract attributes and relationships. It describes the dataset as a resource for training and evaluating scientific-document models, while the appendix overview lists further benchmark and experiment details.

6 Conclusion

  • DocGenome uses DocParser to extract structured attributes and relationships between units in scientific documents.Source: §6 · p. 9
  • Its task coverage, logicality, diversity, and correctness make DocGenome a resource for training scientific-document models and evaluating their capabilities.Source: §6 · p. 9

A Overview of Appendix

  • The appendix overview directs readers to details on limitations and accessibility, annotation explanations, statistical distributions, quality assurance, and task examples.Source: A Overview of Appendix · p. 14

B.1 Limitations

DocGenome is intended to support intelligent document processing and MLLM evaluation. The authors identify the effect of its entity-relationship annotations on scientific-document understanding as a question for future work.

B.1 Limitations

  • DocGenome is intended to support intelligent document processing and effective evaluation of MLLMs on document-understanding tasks.Source: §B.1 · p. 14
  • Although DocGenome annotates six categories of entity relationships, their impact on large models’ scientific-document understanding remains to be explored.Source: §B.1 · p. 14
  • The authors plan to explore the role of entity relationships in scientific-document understanding in future work.Source: §B.1 · p. 14

B.2 Dataset Accessibility

The paper documents DocGenome’s access, preservation, licensing, and dataset statistics, and provides annotation-category explanations. It reports that the collection comes from arXiv open-access papers and describes its document-length and discipline distributions.

B.2 Dataset Accessibility

  • The dataset website documents metadata, format details, and visualizations, and the paper provides a download link.Source: §B.2 · p. 14
  • The authors report extensive dataset statistics and quality checks for both DocGenome-train and DocGenome-test.Source: §B.2 · p. 14
  • The authors state that they will back up the dataset and monitor its availability to support long-term preservation.Source: §B.2 · p. 14
  • DocGenome is released under a CC BY 4.0 license, and the authors report obtaining DOI 10.5281/zenodo.11488587.Source: §B.2 · p. 14
  • The authors say the dataset uses arXiv open-access papers and contains no personally identifiable information or offensive content.Source: §B.2 · p. 14

C Annotation Explanations

  • The annotation table describes DocParser’s layout categories; “others” and “reference” are not used, with indices 6 and 11, respectively.The annotation index corresponds to the category index in the attribute list.Source: Table A.1 · p. 15, C Annotation Explanations · p. 15

D More Statistical Distributions of DocGenome

The appendix reports diversity in DocGenome’s discipline and page-length distributions. Physics, Computer Science, and Mathematics make up major components, while the page-count figures show a broad range of document lengths.

D More Statistical Distributions of DocGenome

E Details of Quality Assurance for QA Data

QA pairs are generated with discipline-specific guidance and independently cross-verified by faculty reviewers. The appendix also presents GPT-acc prompts and a whole-document annotation example.

E Details of Quality Assurance for QA Data

  • Discipline-specific guidance is added to the general GPT-4V prompt template to generate diverse, relevant QA labels.The template is shown in Figure A.3.Source: E Details of Quality Assurance for QA… · p. 16
  • Faculty reviewers rate QA-pair correctness and relevance on a 0–3 confidence scale, then re-analyze and update pairs when their judgments conflict with automated correctness.The scale ranges from confident and accurate (3) to confident the pair is wrong or irrelevant (0).Source: E Details of Quality Assurance for QA… · p. 16

F Prompt Design for GPT-acc

G Examples in Document-level Annotation from DocGenome

H Examples of Tasks in DocGenome-test

The appendix illustrates DocGenome-test tasks and document annotations, alongside visualizations of discipline distribution and QA evaluation prompts.

H Examples of Tasks in DocGenome-test

Related papers

Loading 2406.11633v2…