Source-linked AI summary
OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations
Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongying Tu, Conghui He
TL;DR
Existing document-parsing benchmarks provide limited coverage and simplified evaluation, making fair comparison across document types and methods difficult. OmniDocBench introduces a diverse, richly annotated benchmark with multi-level evaluation, and its experiments reveal different strengths across pipelines and VLMs.
Problem
Existing benchmarks underrepresent document diversity and use limited annotations, output-format handling, and metrics for page-level parsing evaluation.
Method
OmniDocBench combines pages from 9 document types with comprehensive annotations and end-to-end, task-specific, and attribute-based evaluation.
Results
Pipeline tools perform well on academic papers and financial reports, VLMs generalize better to slides and handwritten notes, and pipeline tools outperform VLMs on newspapers.
Takeaways & Limitations
OmniDocBench enables systematic and fair assessment of document-parsing methods while supporting targeted model optimization through task-specific and attribute-level evaluations.
Abstract
from arXiv · showhide
Document content extraction is a critical task in computer vision, underpinning the data needs of large language models (LLMs) and retrieval-augmented generation (RAG) systems. Despite recent progress, current document parsing methods have not been fairly and comprehensively evaluated due to the narrow coverage of document types and the simplified, unrealistic evaluation procedures in existing benchmarks. To address these gaps, we introduce OmniDocBench, a novel benchmark featuring high-quality annotations across nine document sources, including academic papers, textbooks, and more challenging cases such as handwritten notes and densely typeset newspapers. OmniDocBench supports flexible, multi-level evaluations--ranging from an end-to-end assessment to the task-specific and attribute--based analysis using 19 layout categories and 15 attribute labels. We conduct a thorough evaluation of both pipeline-based methods and end-to-end vision-language models, revealing their strengths and weaknesses across different document types. OmniDocBench sets a new standard for the fair, diverse, and fine-grained evaluation in document parsing. Dataset and code are available at https://github.com/opendatalab/OmniDocBench.
1. Introduction
Document parsing supports structured data needs for LLMs and RAG systems, but existing benchmarks do not fairly evaluate diverse document types, formats, and parsing paradigms. OmniDocBench addresses these gaps with diverse data, multi-level evaluation, and broad benchmarking.
- Document parsing extracts structured, machine-readable content from PDFs for LLMs and retrieval-augmented generation systems.
- Pipeline systems decompose parsing into layout analysis, OCR, formula or table recognition, and reading-order estimation, whereas VLMs output structured representations end to end.
- Existing benchmarks are limited by narrow document diversity, inconsistent text-similarity metrics, and insufficiently fine-grained evaluation of formulas, tables, and other content.
- OmniDocBench includes pages from 9 document types, ranging from textbooks to newspapers, with automated annotation, manual verification, and expert review.
- The benchmark supports end-to-end, task-specific, and attribute-based evaluation while comparing pipeline tools and VLMs across document types and content structures.
2. Related Work
Prior document-parsing research separates specialized pipeline modules from end-to-end VLM approaches, while existing benchmarks remain narrow in data, annotation, and evaluation design. These limitations motivate a diverse page-level benchmark with aligned metrics and detailed annotations.
- Pipeline-based Document Content Extraction: Pipeline-based systems combine specialized models for layout detection, OCR, formula recognition, and table recognition, often producing structured formats such as Markdown, JSON, or HTML.
- End-to-End Vision-Language Models: VLMs perform document parsing end to end and can handle diverse or unseen document types without specialized data fine-tuning.
- Benchmarks for Document Content Extraction: Existing benchmarks often evaluate isolated modules or limited page-level sets, overlooking structural and semantic information in full-page document extraction.
- Benchmarks for Document Content Extraction: Page-level benchmarks are constrained by limited data size, language, document types, annotations, output-format alignment, and specialized metrics.
3. OmniDocBench Dataset
OmniDocBench is constructed from visually diverse document pages and uses comprehensive, multi-stage annotation procedures. Its dataset combines broad page coverage, detailed layout and attribute labels, and quality checks for difficult content.
- Dataset Construction: The dataset construction framework combines diverse data acquisition, intelligent pre-annotation, manual refinement, and expert quality checks.
- Data Acquisition: More than 200,000 initial PDF documents were collected, visually clustered with ResNet-50 and Faiss, and sampled into 6,000 diverse pages for annotation.
- Annotation Diversity: Layout annotations include bounding boxes for 19 region categories, reading order, and 9 bbox-level attribute labels spanning text and tables.
- Annotation Quality: Automatic detection and recognition models provide pre-annotations, annotators correct them, and three researchers review unrenderable formula and table elements.
- Page Diversity: OmniDocBench contains 981 PDF pages across 9 types, with page-level attributes covering language, column layout, blurred scans, watermarks, and colored backgrounds.
4. OmniDocBench Evaluation Methodology
OmniDocBench uses an end-to-end evaluation pipeline that extracts elements from generated Markdown, matches predictions with ground truth, and calculates unified metrics. Its matching procedure addresses paragraph splitting and syntactic variation.
- Evaluation Pipeline: The evaluation pipeline contains extraction, matching, and metric-calculation modules for unified document-parsing assessment.
- Preprocessing: Generated Markdown is preprocessed by removing images and leading tags and standardizing repeated characters before element extraction.
- Elements Extraction: Element extraction uses regular expressions in a prescribed order so LaTeX tables, HTML tables, and other elements do not interfere with one another.
Formula
OmniDocBench evaluates document parsing through extraction, matching, and metric-calculation modules. Its matching handles paragraph alignment and ignored components, while task-specific metrics assess text, tables, formulas, reading order, and layout.
- Formula: The evaluation pipeline combines extraction, matching, and metric calculation to produce unified document-parsing results.The modules are designed to automate fair and reliable evaluation.
- 4.2. Matching Algorithm: Adjacency Search Match merges or splits ground-truth and predicted paragraphs to maximize normalized-edit-distance similarity.It first accepts sufficiently similar pairs, then applies fuzzy subset matching and iterative adjacent-paragraph merging.
- 4.2. Matching Algorithm: The benchmark reports comprehensive results for text, formula, table, and reading-order extraction against ground-truth comparisons.The overall evaluation is organized in Table 2.
- 4.2. Matching Algorithm: End-to-end text recognition is evaluated across nine PDF page types using edit distance.Table 3 organizes performance by page type.
- 4.3. Metric Calculation: Reading-order evaluation compares column-layout types, while layout and table components receive dedicated evaluations.Tables 4, 5, 6, and 7 cover page attributes, column layouts, layout detection, and table recognition.
- 4.2. Matching Algorithm: Headers, footers, page numbers, page footnotes, and captions participate in matching but are excluded from metric calculations.The ignore logic addresses inconsistent model output standards.
- 4.3. Metric Calculation: Pure text uses normalized edit distance, while tables use HTML-based TEDS and normalized edit distance.These metrics are averaged at the sample level for final scores.
- 4.3. Metric Calculation: Formula evaluation uses CDM, normalized edit distance, and BLEU, while reading order uses normalized edit distance.Reading-order calculation excludes tables, images, and ignored components.
5. Benchmarks
Benchmark results show complementary strengths across parsing approaches and document conditions. Specialized tools lead many component tasks, while general VLMs generalize better to some specialized pages and visual degradations, but struggle with dense newspapers and several recognition conditions.
- 5.1. End-to-End Evaluation Results: Pipeline tools lead text, formula, and table recognition, while Qwen2-VL and GPT4o remain competitive among general VLMs.Nearly all algorithms score higher on English than Chinese pages.
- 5.1. End-to-End Evaluation Results: Pipeline tools perform well on academic papers and financial reports, whereas general VLMs generalize better to slides and handwritten notes.Most VLMs fail on newspapers, while pipeline tools perform significantly better there.
- 5.1. End-to-End Evaluation Results: VLMs such as InternVL2 and Qwen2-VL show greater robustness to fuzzy scans, watermarks, and colorful backgrounds.MinerU remains competitive among pipeline tools because of layout segmentation and preprocessing.
- 5.1. End-to-End Evaluation Results: High-density documents challenge VLMs because of input-resolution and token-length limitations, while layout-based segmentation helps pipeline tools process components individually.Layout-aware designs and domain-specific fine-tuning are identified as promising directions.
- 5.2. Single Task Evaluation Results: DocLayout-YOLO significantly outperforms other layout detectors and contributes to MinerU’s strong overall performance.Other methods perform well on books and academic literature but struggle with more diverse formats.
- 5.2. Single Task Evaluation Results: RapidTable achieves the highest overall table-recognition scores for language diversity and remains stable across frame types.General-purpose VLMs perform lower but consistently, while StructEqTable excels on noframe tables and rotation robustness.
- 5.2. Single Task Evaluation Results: PaddleOCR outperforms competing OCR tools, while general VLMs struggle with rotated text and mixed-language scenarios.Formula recognition scores are 86.8% for GPT-4o, 86.6% for Mathpix, and 85.0% for UniMERNet.
6. Conclusion
OmniDocBench addresses the shortage of diverse, realistic document-parsing benchmarks with comprehensive annotations and flexible evaluation. Its task-specific and attribute-level analyses support systematic assessment and targeted model optimization.
- 6. Conclusion: OmniDocBench combines diverse page types, comprehensive annotations, and a flexible evaluation framework for fair document-parsing assessment.The benchmark is intended to provide systematic insights for advancing document parsing.
- 6. Conclusion: Task-specific and attribute-level evaluations enable targeted optimization of document-parsing models.The supported evaluation scope includes the benchmark’s annotated document characteristics.
I. More End-to-End Evaluation Results
Additional end-to-end analyses examine language, table, text, and annotation attributes across a broad benchmark. Results identify accuracy losses from rotation, Chinese or mixed-language content, and selected table conditions, with model-specific robustness differences.
- Table Attributes: Most models recognize English tables more accurately than Chinese tables, while full-frame, noframe, and special-condition tables reduce performance.Table rotation significantly affects all models; merged cells mainly test recognition breadth, whereas colored backgrounds test robustness.
- Table Attributes: Pipeline tools tolerate challenging merged-cell tables better, but colored backgrounds can reduce their recognition accuracy.Several VLMs perform worse on merged cells, while colored backgrounds have limited effect on their table recognition.
- Text Attributes: Almost all models recognize Chinese text less accurately than English text, and MinerU and Marker decline further on mixed Chinese-English content.The text-recognition passage attributes MinerU’s decline to its PaddleOCR module.
- Dataset Attributes: OmniDocBench contains 981 pages spanning nine PDF-page types, four layouts, three languages, and three visual-degradation issues.Examples and attribute counts are documented in the supplementary tables and figures.
- Annotation Categories: The benchmark provides bounding-box annotations for 15 block-level and four span-level categories, plus three page-interference categories.Span annotations are nested within block annotations, and interference boxes mask affected page regions.
- Annotation Categories: Recognition annotations cover every category except figures, with formulas in LaTeX, tables in HTML and LaTeX, and other categories in plain text.These formats support category-specific recognition evaluation.
- Text Attributes: Text attributes include language, background color, and rotation across 23,010 block-level annotations.These attributes are identified as factors that may influence OCR accuracy.
- Table Attributes: Table attributes cover language, frame type, merged cells, colorful backgrounds, formulas, and rotation.These attributes are annotated because they may influence table-recognition accuracy.
III. Discussion on Model Predictions
Model performance varies substantially across document types, page conditions, layouts, text attributes, and table structures. Pipeline tools and expert models are stronger on common documents and difficult tables, while VLMs generalize better to uncommon formats and some special page conditions.
- Pipeline tools and expert models outperform VLMs on common documents such as academic papers and on tables with merged cells.
- VLMs generalize better to uncommon PDF types such as slides and exam papers, while pipeline tools generalize weakly to handwriting notes.
- Pipeline and VLM weaknesses differ under special page conditions: pipelines are less robust to watermarks, whereas VLMs may miss dense-page content or hallucinate on hard-to-recognize pages.
- Multi-column layouts expose different failures: InternVL2 performs poorly on double- and three-column pages, while MinerU may merge columns and lose reading-order accuracy on complex layouts.
- Table attributes alter model behavior: RapidTable handles three-line structures well, StructEqTable recognizes most rotated-table content, and formula-containing tables favor Qwen2-VL-7B over InternVL2-8B.
V. More Details on Methods
The evaluation method reduces differences caused by inconsistent parsing conventions and presents a broad collection of document, layout, text, and table examples. The supplementary materials illustrate both dataset diversity and model behavior across these conditions.
- Ignore handling minimizes evaluation effects from differing conventions for retaining headers, footers, captions, and footnotes.
- The supplementary examples cover academic papers, books, textbooks, notes, magazines, financial reports, newspapers, example papers, and slides.
- Additional examples represent layout types, special page issues, table frames, text languages, rotated text, and tables containing formulas.
- Good and bad model outputs are shown for document types including academic papers, books, exam papers, magazines, newspapers, handwriting notes, financial reports, slides, textbooks, and fuzzy scans.
- The supplementary material also compares outputs for single-, double-, and three-column pages, complex layouts, colorful backgrounds, watermarks, and multiple table structures.