PaperlayerView exact revision on arXiv

Source-linked AI summary

How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean, Degraded, and Real-World Settings

Zhiheng Li, Zongyang Ma, Jiaxian Chen, Jianing Zhang, Zhaolong Su, Yutong Zhang, Zhiyin Yu, Ruiqi Liu, Xiaolei Lv, Bo Li, Jun Gao, Ziqi Zhang, Chunfeng Yuan, Bing Li, Weiming Hu

arXiv:2605.07492v2cs.CV

TL;DR

Existing document-parsing benchmarks are labor-intensive to extend across document types and domains. PureDocBench links generated document sources to annotations and matched image tracks, then evaluates 58 models. Scores vary across acquisition conditions, with average losses larger on Real than Digital images.

  • Problem

    Extending document benchmarks to new types or domains often requires repeating labor-intensive collection and annotation.

    Source: §1 · p. 2

  • Method

    PureDocBench links generated HTML/CSS documents to annotations and evaluates 58 models on matched Clean, Digital, and Real versions of each page.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    Average Overall losses across 58 models are 3.34 points on Digital and 11.61 points on Real, relative to Clean.

    Source: §3.2 · p. 5

  • Takeaways & Limitations

    Results show room for improvement across parsing components and image conditions, while compact specialists can match or outperform much larger general models.

    Source: §4 · p. 8

    The adopted evaluation metrics need refinement for semantic omissions and structural consistency, and HTML rendering has limited coverage of handwriting and historical books.

    Source: §4 · p. 9

Abstract

from arXiv · show

Document parsing models are improving rapidly, but existing benchmarks are still insufficient for a comprehensive evaluation of their capabilities. We introduce PureDocBench, a benchmark with 10 domains, 66 subcategories, and 1,475 pages. Each page has matched Clean, Digital, and Real versions, covering clean renders, simulated degradation, and real capture or sharing conditions. The resulting 4,425 images have source-derived annotations for text, formulas, tables, and reading order, checked through automated validation and human review. Keeping page content and annotations fixed allows direct comparisons across image conditions. We evaluate 58 models, including pipeline-based parsers, end-to-end parsers, and general vision-language models. The best three-track average score is 79.69 out of 100, and the mean across models is 63.84. Scores drop by 3.34 points on Digital and 11.61 points on Real on average, relative to Clean. TeleOCR leads on Clean, GLM-5.3-Flash on Digital, and Gemini 3.6 Flash on Real. The benchmark dataset, source documents, generation tools, and evaluation code are publicly available at https://github.com/zhihengli-casia/PureDocBench.

1 INTRODUCTION

Existing document benchmarks are valuable but laborious to extend to new types and domains. PureDocBench pairs a reusable source-linked benchmark with evaluation of 58 models, revealing broad accuracy and robustness differences.

  • Extending existing benchmarks to new document types or domains often requires repeating labor-intensive collection and annotation, while PureDocBench's pipeline is reusable.Source: §1 · p. 2
  • Best Avg3 is 79.69, mean score is 63.84, and the best-to-worst spread is 50.2 points across 58 evaluated models.Source: §1 · p. 2
  • PureDocBench compares 58 specialist parsers and general VLMs across three image tracks, evaluating overall accuracy, robustness, and individual parsing components.Source: §1 · p. 2
  • The benchmark covers 1,475 pages across 10 domains and 66 subcategories, with source-linked annotations, matched image tracks, and released resources for reproducible evaluation and extension.Source: §1 · p. 2

2 PUREDOCBENCH

PureDocBench generates document pages from HTML/CSS sources and links their images to annotations, with matched image tracks for comparisons. It positions this source-traceable, human-reviewed benchmark alongside resources covering component recognition, document understanding, and full-page parsing.

2.1 DATA CONSTRUCTION

  • Each of 1,475 pages has Clean, Digital, and Real versions sharing ground truth, enabling acquisition comparisons while varying image conditions.Digital degradation uses ten scene templates; Real degradation uses four capture or sharing chains.Source: §2.1 · p. 2
  • LLMs generate HTML/CSS and extract annotations from the same source, while human screening, deterministic checks, and cross-review validate all 1,475 pages.A browser renders high-resolution PNGs; annotations include text, table HTML, formulas, and reading order.Source: §2.1 · p. 3
  • Retained sources support annotation inspection and regeneration, and the validation pipeline can be reused to extend pages and document types.Source: §2.1 · p. 3

2.2 COMPARISON WITH RELATED BENCHMARKS

  • PureDocBench combines generated HTML/CSS sources, human-reviewed annotations, and matched clean, digital, and real images, complementing authentic-document and multi-page benchmarks.The comparison spans component recognition, document understanding, and full-page parsing.Source: Table 1 · p. 4, §2.2 · p. 5

3 EXPERIMENTS

The evaluation compares 58 models across three image conditions, measuring overall parsing accuracy, robustness, component strengths, and concrete failure cases.

3.1 SETUP

  • The 58-model comparison spans 13 pipeline/multi-stage specialists, 19 end-to-end specialists, and 26 general VLMs, using Overall, Avg3, and separately reported reading-order scores.Overall averages TextEdit, FormulaCDM, and TableTEDS; Avg3 averages the three track Overalls.Source: §3.1 · p. 5

3.2 MAIN RESULTS

  • 79.69 (GLM-5.3-Flash) is the best Avg3 and 63.84 is the model mean; average Overall losses are 3.34 points on Digital and 11.61 on Real.Pipeline specialists lose 5.17/14.23 points on Digital/Real, E2E specialists 4.14/12.99, and general VLMs 1.84/9.30.Source: §3.2 · p. 5, Table 2 · p. 6, Figure 3 · p. 7
  • MinerU2.5-Pro (1.2B) scores Avg3 70.07, close to Kimi K2.6 (70.10); WeVisDoc-2B exceeds Qwen3.5-27B by 4.12 points with 7.4% of its parameters.WeVisDoc-4B scores 75.57, exceeding Qwen3.5-122B-A10B (74.11) by 1.46 points.Source: §3.2 · p. 5

3.3 COMPONENT ANALYSIS

  • Across three-track means, E2E specialists lead mean Text/Reading scores (66.18/57.44), while general VLMs lead Formula/Table (62.48/70.07).Individual leaders are Qwen3.8-27B for Text (78.00), Gemini 3.6 Flash for Formula (77.81), GLM-5.3-Flash for Table (87.35), and TeleOCR for Reading (75.14).Source: §3.3 · p. 5, Figure 4 · p. 8

3.4 CASE STUDY

  • Clean-track examples include an omitted Social label and 25% weight, a changed bill-of-lading total, an omitted $3,845 charge, and formula or symbol substitutions.The medical-bill omission is localized to the first table’s total row; the same amount remains in the following table.Source: §3.4 · pp. 7–8
  • On Digital, Gemini-3.1-Pro changes a clearly readable hotel-dashboard revenue-growth percentage from +11% to 112%, while paired Clean and Real predictions preserve +11%; on Real, MinerU2.5-Pro omits “Hazelnut” from a creased-menu ingredient list.The corresponding Clean and Digital predictions preserve the complete ingredient list. Figure 5 shows omissions, numerical errors, and failures in table structure, formulas, and symbol recognition across the three tracks.Source: §3.4 · pp. 7–8

4 CONCLUSION AND FUTURE WORK

PureDocBench provides matched, source-linked document images and finds that parsing still has room to improve across components and image conditions. The authors identify limits in evaluation metrics and HTML-rendered coverage, and release resources to support reproducible comparisons.

4 CONCLUSION AND FUTURE WORK

  • Across 58 models, results show room for improvement across parsing components and image conditions, while compact specialists can match or outperform much larger general models.The benchmark and its document sources, generation tools, and evaluation code are released to support reproducible comparisons and further research.Source: §4 · p. 8
  • Parsing failures include omissions, altered values, missing table cells, and formula-symbol errors, affecting content completeness, numerical accuracy, and document structure.Examples include a subscript rewritten as division and Greek symbols read as digits.Source: Figure 5 · p. 9
  • The evaluation framework’s metrics need refinement for semantic omissions and structural consistency, while HTML rendering has limited coverage of handwriting and historical books.The authors say these settings require additional real-document data.Source: §4 · p. 9
  • Applications involving sensitive information should include privacy safeguards and human verification.Source: §4 · p. 10

A BENCHMARK COMPARISON: SCOPE, VERSIONS, AND EVIDENCE

The comparison organizes document resources by task scope, annotation provenance, and paired image variants. It places PureDocBench among benchmarks spanning component recognition, full-document parsing, acquisition robustness, and broader document understanding.

A BENCHMARK COMPARISON: SCOPE, VERSIONS, AND EVIDENCE

  • The comparison distinguishes text, table structure, formula recognition, and reading order, while treating layout detection and question answering as complementary dimensions.Its checkmarks indicate evaluated tasks, restricted marks indicate subtasks or subsets, and dashes indicate unreported dimensions; these symbols summarize scope, not metric quality.Source: A BENCHMARK COMPARISON: SCOPE,… · p. 16
  • Prior component datasets establish source-derived supervision precedents, while full-document and task-focused suites cover differing targets, annotation processes, and evaluation scopes.For example, PubTabNet pairs table crops with HTML, while READoc derives Markdown targets from multiple source formats.Source: A BENCHMARK COMPARISON: SCOPE,… · pp. 16–18
  • Document understanding benchmarks evaluate QA, retrieval, reasoning, or grounded answers, which complement but do not establish downstream performance from parsing scores.The comparison also covers multilingual, specialized-content, synthetic-supervision, and controlled-evaluation resources with distinct scope boundaries.Source: A BENCHMARK COMPARISON: SCOPE,… · pp. 17–18
  • Matched robustness resources include DocPTBench, Real5-OmniDocBench, and MDPBench, but their page counts and acquisition protocols differ.PureDocBench pairs all 1,475 pages across three tracks; DocPTBench physically captures a 100-page subset, and Real5 applies five physical conditions to its pages.Source: A BENCHMARK COMPARISON: SCOPE,… · pp. 17–18
  • PureDocBench pairs 1,475 generated pages across clean, digital, and real tracks and retains their common HTML/CSS sources for inspection and regeneration.The comparison frames this as building on prior matched-acquisition datasets while combining source retention and matched tracks.Source: A BENCHMARK COMPARISON: SCOPE,… · p. 18

B ADDITIONAL CASE-STUDY EVIDENCE

The case studies illustrate parsing errors across clean, digital, and real document images, including omissions, incorrect associations, and altered numerical or structural details. Examples from Dolphin-v2 span multiple document types and error forms.

B ADDITIONAL CASE-STUDY EVIDENCE

  • Clean-track examples include an omitted laboratory sidebar, a skill score copied from a neighboring label, altered paragraph numbering and percentages, and scientific values shifted by orders of magnitude.Other examples show a percent sign misread as a digit in a formula and a code identifier losing its separator.Source: Figure 6 · p. 19, Figure 7 · p. 20, Figure 8 · p. 21

Dolphin-v2

C REAL-TRACK READABILITY AUDIT

Excluding 83 unreadable Real images increases absolute scores but leaves model rankings largely unchanged in the 51-model sensitivity analysis. The audit identifies these images through original-resolution human review and notes that Clean and Digital counterparts remained readable.

C REAL-TRACK READABILITY AUDIT

  • Two annotators reviewed candidate images at original resolution with zoom and reconciled decisions, identifying 83 Real images (5.63%) whose text could not be reliably verified against transcription.Their Clean and Digital counterparts remained readable; the 1,024-pixel setting was only a geometric screening diagnostic.Source: C REAL-TRACK READABILITY AUDIT · p. 29
  • The sensitivity analysis excludes the 83 flagged images only from the Real track, retaining 1,392 Real pages while Clean and Digital each retain all 1,475 pages.It compares full and filtered evaluations for 51 models, recomputing Avg3 with the filtered Real score.Source: C REAL-TRACK READABILITY AUDIT · p. 29, Table 3 · p. 30
  • ρ=0.9983 for Real and ρ=0.9991 for Avg3; excluding 83 unreadable images leaves top-ten membership unchanged, with maximum rank changes of two positions.The mean Real score gain is 2.32 points and the Avg3 gain is 0.77 points; Gemini 3.6 Flash remains the Real leader and GLM-5.3-Flash the Avg3 leader.Source: C REAL-TRACK READABILITY AUDIT · p. 29, Table 3 · p. 30
  • Real recaptures of long pages can obscure fine text that remains readable in Clean, including emergency contacts and a high-voltage warning.Source: Figure 16 · p. 29

D FULL COMPONENT RESULTS

Tables 4–6 report component-level results for 58 models across Clean, Digital, and Real tracks, with accuracy scores and normalized edit distances.

D FULL COMPONENT RESULTS

  • Tables 4–6 report Clean, Digital, and Real component results for 58 models.The reported components are TextEdit, Formula CDM, Table TEDS, Overall, and Reading Edit.Source: Table 4 · p. 31, Table 5 · p. 32, Table 6 · p. 33
  • Text and Reading Edit are normalized edit distances, while Formula CDM, Table TEDS, and Overall use a shared 0–100 scale.Source: Table 4 · p. 31
  • Bold and underlining identify the best and runner-up results in each table.Source: Table 4 · p. 31

E OMNIDOCBENCH ANNOTATION ERROR EXAMPLES

The OmniDocBench v1.5 examples show several kinds of annotation discrepancy, from text errors to structural and category mismatches. Such discrepancies can affect evaluation scores.

E OMNIDOCBENCH ANNOTATION ERROR EXAMPLES

F DEGRADATION PARAMETER DETAILS

The degradation design separates synthetic digital scenes, physical or screen-mediated recaptures, and controlled ablations while keeping annotations linked to the same clean pages. The figures document operation parameters and capture scenarios used for sensitivity analysis.

F DEGRADATION PARAMETER DETAILS

  • Released Clean, Digital, and Real tracks map each image back to the same clean-page annotation, so score differences compare image-condition robustness rather than content changes.The digital track uses a merged selection of one scene version per clean page, with explicit provenance mapping.Source: F DEGRADATION PARAMETER DETAILS · p. 35
  • Ten digital scene profiles combine degradation operations, while leaving the underlying HTML source and ground-truth ordering unchanged.The released digital scenes are balanced at 147–148 pages each.Source: F DEGRADATION PARAMETER DETAILS · p. 35
  • The Real track uses physical or screen-mediated recaptures, with filenames and mappings recording acquisition conditions and linking images to their clean-page annotations.Encoded conditions include source, paper state, lighting, and view geometry.Source: F DEGRADATION PARAMETER DETAILS · p. 35
  • The ablation generator decomposes degradation into 15 operations across Print, Paper, Capture, and Digital stages, with parameter values at s = 0.7 and s = 1.0.Figure 18 lists the operations and parameter values.Source: F DEGRADATION PARAMETER DETAILS · p. 35, Figure 18 · p. 36
  • Figure 19 summarizes ten capture scenarios through strength ranges, weighted operations, and each scenario’s three dominant degradation sources.Source: F DEGRADATION PARAMETER DETAILS · p. 35, Figure 19 · p. 36

G FULL PER-CATEGORY RESULTS

Per-category results show substantial variation in domain difficulty and which model architectures lead. Tables 7–16 report component scores across all ten domains and three image tracks.

G FULL PER-CATEGORY RESULTS

Loading 2605.07492v2…