PaperlayerView exact revision on arXiv

Source-linked AI summary

Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation

Yuheng Chen, Yufan Chen, Zhuojun Cai, Ruiping Liu, Junwei Zheng, Jiale Wei, Di Wen, Weijia Fan, Kunyu Peng, Jiaming Zhang, Rainer Stiefelhagen

arXiv:2610.20335v1cs.CVcs.AI

TL;DR

Document benchmarks often assess layout, text, reading order, and image description separately, leaving their agreement in a single page parse untested. Read-Parse-Describe defines a unified structured output, and SARA adds spatial guidance to VLM tokens. Experiments report that systems struggle across components, while SARA tuning substantially improves structural parsing for a general VLM.

  • Problem

    Existing benchmarks usually test document layout, text, reading order, and visual description separately rather than requiring them to agree in one page-level parse.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    Read-Parse-Describe defines an ordered element schema for page images, and SARA injects aligned multi-layer spatial features into VLM tokens.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Existing systems tend to fail different parts of the evaluation, while frozen-backbone SARA tuning substantially improves structural parsing for a general VLM.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The benchmark jointly evaluates localized elements, labels, order, text and table content, caption links, and alternative text within a single parse.

    Source: §7 · p. 9

    The benchmark covers open-access ACM scientific PDFs and primarily their first eight pages, and generated accessibility metadata is not deployment-ready without further validation.

    Source: §7 · p. 9

Abstract

from arXiv · show

Document understanding systems are typically evaluated as separate components via layout analysis, optical text recognition, reading-order prediction, and image captioning. This separation leaves scientific figures, captions, and visual semantics weakly connected, and fails to measure whether a model can produce a single page-level parse that is both structurally faithful and semantically accessible. We introduce Read-Parse-Describe, a unified dataset for document parsing with visual element description generation. Given a page image, models are required to output an ordered list of layout elements containing bounding boxes, semantic labels, reading-order indices, recognized textual content, and alternative text descriptions for visual elements. The benchmark is built from 2,402 documents, comprising 18,163 continuous pages, 125,218 layout elements, and 11,338 visual instances. Besides, we further propose a plug-and-play space-awareness enhancer that reuses multi-layer visual features and injects grid-aware spatial guidance into vision-language tokens for layout-aware generation. Experiments across end-to-end VLMs and pipeline parsers show that existing systems still struggle to jointly recover reading order and visual semantics, while our enhanced model achieves strong joint performance. These results highlight the need to move beyond isolated document parsing modules toward unified models that jointly read, parse, and describe complex documents. The new dataset and code will be made publicly available at https://github.com/yufanchen96/RPD

1 Introduction

Existing benchmarks test document understanding components separately, rather than whether a page parse keeps structure, text, captions, and visual meaning consistent. Read-Parse-Describe unifies these fields in one task, while SARA adds spatial guidance for VLM parsing.

1 Introduction

  • Existing benchmarks measure layout, OCR, reading order, or image description separately, leaving their agreement within one page parse untested.This limitation is especially relevant to scientific PDFs, where captions, figures, and equations interact with page structure.Source: §1 · p. 2
  • Read-Parse-Describe requires an ordered page parse with boxes, labels, reading order, recognized text, caption links, and visual descriptions.Its dataset contains 2,402 documents, 18,164 continuous pages, 125,218 layout elements, and 11,338 visual instances.Source: §1 · p. 2
  • SARA reuses intermediate visual features, aligns them to crop-aware spatial grids, and injects them into VLM tokens through a residual adapter.The base VLM is frozen, and the adapter leaves benchmark labels and evaluation unchanged.Source: §1 · p. 2
  • The unified evaluation protocol assesses parsing fidelity and alternative-text generation quality together.The task makes agreement among page elements' attributes part of one auditable parse.Source: §1 · p. 2
  • Crop-aware spatial alignment consistently benefits unified document parsing and alternative-text generation in the reported experiments.The paper also reports that SARA tests a direct architectural response to spatial-alignment failures without changing labels or evaluation.Source: §1 · p. 2

2 Related Work

Prior work has developed specialized resources and models for layout, parsing, document VLMs, and visual description. Read-Parse-Describe positions its contribution at the intersection: a page-level parse that binds visual descriptions to page structure.

2.1 Document Layout and Parsing Datasets

  • Layout and parsing datasets target distinct outputs, from document classes and forms to table structure, layout regions, and reading order.Broader evaluations such as DocVQA [31] and DUDE [32] test document understanding through answers rather than page parses; OmniDocBench [33] moves closer to parsing.Source: §2.1 · pp. 2–3

2.2 Document VLMs and Pipeline Systems

  • Document systems span structure-aware encoders, OCR-free generators, specialized parsers, general VLMs, and pipeline approaches.Examples include LayoutLM [1], Donut [6], Nougat [38], and MinerU [42].Source: §2.2 · p. 3

2.3 Visual Descriptions and Accessibility

  • Accessibility datasets address varied description settings, while Read-Parse-Describe evaluates alternative text as a field linked to the object's caption, position, and reading flow.The related work includes blind-user scenarios in VizWiz [45] [46], visible-text descriptions in TextCaps [47], and context-conditioned alt text in Alt-Text with Context [16].Source: §2.3 · p. 3

3 Read-Parse-Describe Dataset

Read-Parse-Describe treats document understanding as a unified, spatially grounded parse, with visual descriptions attached to their page evidence. Its dataset construction and annotation procedures jointly address cross-field consistency and the auditability of alt-text labels.

3.1 Task Definition

  • Visual elements are often localized as layout categories without requiring their conveyed information to be expressed in a grounded, accessible form.The task is motivated by gaps in logic-aware representations and ordering in scientific papers.Source: §3.1 · p. 3

3.2 Output Schema

  • The target is an ordered list whose elements carry boxes, semantic categories, reading-order indices, recognized content, caption links, and visual descriptions when applicable.Region proposals, order, caption linkage, and descriptions are coupled through their page evidence.Source: §3.2 · pp. 3–4

3.3 Dataset Creation Pipeline

  • The dataset retains the first eight pages of open-access ACM papers, keeps shorter papers in full, and splits papers 90:5:5 after deduplication.Paper-level splitting addresses shared templates and repeated content across adjacent pages.Source: §3.3 · pp. 4–5

3.4 Annotation Pipeline

  • Layout boxes, labels, caption links, and reading-order indices are jointly inspected and corrected; alt text receives rubric scoring, targeted human refinement, and re-scoring.Human refinement targets low-scoring descriptions; the procedure is designed for benchmark consistency, not automatic accessibility deployment.Source: §3.4 · p. 5
  • After refinement, the score-5 rate rises from 66.2% to 98.1%, and a 200-sample audit finds exact human–LLM score agreement on 194 samples.The audited cases were revised samples whose previous scores were below 5.Source: §3.4 · p. 5

4 SARA: Spatial Alignment Residual Adapter

SARA restores crop-level spatial evidence to the decoder's merged visual tokens while preserving the base VLM pathway. Its adapter uses multi-layer features, grid-aware crop alignment, and gated residual injection.

4 SARA: Spatial Alignment Residual Adapter

  • SARA restores crop-level spatial evidence where the decoder receives merged visual tokens, preserving crop-to-token correspondence without ground-truth boxes.The goal is to support decisions about element boundaries, reading order, and caption-object matching.Source: §4 · p. 5
  • The adapter concatenates encoder-layer features, aligns them with grid-aware multi-crop guidance, and injects the aligned features into original tokens through a gated residual connection.This design reuses multi-layer visual features while preserving spatial structure.Source: Figure 2 · p. 6

5 Method

SARA augments the visual-token pathway with crop-aligned spatial guidance while retaining the original token stream. It reuses intermediate visual features, aligns them to merged-token grids, and injects them through a learnable residual gate.

5 Method

  • SARA reuses intermediate visual states, aligns them to the native crop grid, and injects the guidance through a learnable residual gate.The design preserves the pretrained visual-language pathway while training determines how much spatial correction to apply.Source: §5 · p. 6
  • Selected visual states are concatenated across channels and projected to the merger dimension before crop-aware alignment.The projection uses LayerNorm and GELU; the selected layers are {2, 5, 8, 11}.Source: §5 · p. 6
  • For each crop, guidance is reshaped to its two-dimensional grid and average-pooled to the merged-grid resolution.Flattened crop features are concatenated into G′, matching the length and dimension of the original merged visual tokens.Source: §5 · p. 6
  • The enhanced tokens add the gated guidance to the original visual tokens, while the decoder retains the shared output schema.SARA changes only the visual-token pathway, not benchmark labels, target schema, or evaluation metrics.Source: §5 · p. 6
  • SARA is not intended to replace document-specific supervision or table-aware decoding.The residual form keeps the original visual-token stream intact and lets training learn the aligned guidance.Source: §5 · p. 7

6 Experiments

Experiments evaluate diverse VLMs, document-specialized models, and pipeline parsers under a shared page-level schema. Results show complementary strengths across structural parsing and visual descriptions, with SARA improving structural fields while some gains vary by component, feature layer, and training-data scale.

6.1 Settings and Metrics

  • The shared evaluation covers general-purpose VLMs, document-specialized models, and pipeline parsers, converting outputs to one schema before scoring.F1 uses category-matched IoU ≥0.5; reading order uses normalized edit-distance similarity, OCR uses normalized character-level similarity, and descriptions use METEOR and BERTScore.Source: §6.1 · p. 7

6.2 Main Results

  • Table 2 shows complementary strengths: specialized parsers excel on some fields, while SARA improves a frozen general-purpose VLM’s structural fields and maintains competitive visual-description quality.The authors frame the table as a contract diagnostic rather than a single-model leaderboard.Source: §6.2 · p. 7, Table 2 · p. 8
  • Fine-tuned Dolphin-v2 reaches 0.9826 F1, 0.9588 mIoU, and 0.9857 RO; DeepSeek-OCR leads METEOR at 0.3998, while GPT-5.6 leads BERTScore at 0.9004.Among proprietary zero-shot models, GPT-5.6 has the highest F1, RO, and description scores, while Gemini 3.6 Flash has the highest mIoU.Source: §6.2 · p. 7
  • Pipeline parsers can retain strong low-level fields but leave the unified parse incomplete; MinerU has 0.9671 OCR accuracy and 0.9151 mIoU, but 0.6757 RO and no visual alt-text.The text identifies fields needing additional modules or tighter integration for a complete accessible page parse.Source: §6.2 · p. 7
  • SARA reaches 0.9189 F1, 0.8919 mIoU, 0.9636 OCR accuracy, and 0.9774 RO against the Qwen3.5-2B full fine-tuning baseline.It trains 83.99M parameters with the base VLM frozen; its alt-text scores are 0.3834 METEOR and 0.8946 BERTScore.Source: §6.2 · pp. 7–8

6.3 Ablations and Analysis

  • On Qwen3.5-2B, grid alignment alone modestly improves over full fine-tuning, projector-only training underperforms it, and the combined configuration gives the largest gains across four structural metrics.Adapter rows keep the backbone frozen.Source: §6.3 · pp. 8–9, Table 3 · p. 9
  • Early-to-mid feature layers {2,5,8,11} outperform shallow-only and late-layer selections; SARA improves localization on both tested backbones, while reading-order and description effects vary.The ablations characterize SARA primarily as a spatial-structure adapter rather than a general-purpose language-generation module.Source: §6.3 · p. 9, Table 4 · p. 9
  • SARA trails full fine-tuning on all four structural metrics with 10% training data, gives mixed results at 30%, and gains a structural advantage at 50% and 100%.The complete comparison is reported in Appendix E.Source: §6.3 · p. 9

7 Conclusion

Read-Parse-Describe unifies scientific-document parsing in a page-level benchmark, while SARA applies spatial adaptation to the same output contract. Experiments show varied system failures and improved structural parsing with frozen-backbone SARA tuning for a general VLM.

7 Conclusion

  • Frozen-backbone SARA tuning substantially improves structural parsing for a general VLM, while evaluated systems tend to fail different parts of the task.The benchmark jointly evaluates localized elements, labels, reading order, text and table content, caption links, and alternative text within one ordered parse.Source: §7 · p. 9
  • The benchmark covers open-access ACM scientific PDFs and primarily the first eight pages, underrepresenting late appendices and references.Generated accessibility metadata is not deployment-ready without further validation.Source: §7 · p. 9

A Dataset Details

This appendix details the dataset’s label taxonomy, page curation, and alt-text annotation process. The benchmark keeps scholarly elements distinct and prioritizes early pages with dense multimodal content.

A.1 Label Taxonomy and Corpus Profile

  • The taxonomy uses 23 labels to keep reading-order, reflow, caption-linkage, and auditability targets distinct.Headings, paragraph fragments, captions, equations, code blocks, metadata, references, lists, and visual objects remain separately labeled.Source: §A.1 · p. 15
  • The benchmark contains 2,402 curated scientific documents and 18,164 continuous pages.The domain chart summarizes a broader pool of 2,151 ACM articles from 2020–2024, not the benchmark itself.Source: §A.1 · p. 15

A.2 Page Curation and Element Distributions

  • The first-eight-page rule preserves early-paper regions where abstracts, methods, figures, tables, captions, equations, and author metadata interact, while later reference-heavy pages are underrepresented.The stated goal is to increase multimodal-content density while retaining a continuous page sequence.Source: §A.2 · p. 15
  • The appendix’s distribution tables describe visual instances by subject category and image type, and layout groups by annotation-budget allocation.The tables offer complementary views of disciplinary context, visual form, and the balance among document-structure elements.Source: §A.2 · pp. 15–16

B Alt-text Annotation Details

  • Alt-text annotation separates generation, rubric-based scoring, targeted human refinement, and manual audit to make description assumptions inspectable.Low-scoring descriptions are revised and re-scored under the same rubric; the score distribution compares the same 11,338 visual instances before and after refinement.Source: B Alt-text Annotation Details · pp. 16–17

C Evaluation Metric

The evaluation suite measures physical localization, logical reading order, textual and tabular accuracy, and accessibility descriptions. Its metrics distinguish detection and boundary quality from sequencing, transcription, table structure, and semantic description matching.

C Evaluation Metric

  • The evaluation separates localization, sequencing, and textual or tabular accuracy because correct detection can coexist with ordering errors, and fluent text with misplaced boxes.This multi-dimensional suite also includes accessibility metadata evaluation.Source: C Evaluation Metric · p. 17

1. Physical Layout Analysis: F1-score and mIoU

  • Layout evaluation combines category-sensitive F1 for detections with mIoU for geometric localization accuracy.F1 true positives require category agreement and overlap above a predefined threshold; mIoU measures box overlap over matched instances.Source: §1 · p. 19

2. Logical Reading Order: Normalized Edit Distance (NED)

  • Reading order is scored by normalized Levenshtein edit distance between predicted and ground-truth block sequences, with higher scores indicating more faithful logical flow.The edit distance counts insertions, deletions, and substitutions, and normalization supports comparisons across documents with different numbers of blocks.Source: §2 · p. 19

3. Textual and Tabular Accuracy: ED and TEDS

  • OCR accuracy uses character-level edit distance against reference transcription, while TEDS scores both table structure and cell content using tree representations.TEDS penalizes structural errors such as merged or split cells as well as textual mistakes.Source: §3 · pp. 19–20

4. Accessibility Metadata Evaluation: METEOR Score

  • Alt-text quality for figures and tables is evaluated with METEOR, which is intended to capture semantic nuances beyond exact n-gram overlap.Its described scoring uses precision and recall with greater emphasis on recall, plus a penalty for fragmented alignments.Source: §4 · p. 20

D Evaluation Details

All systems are mapped to a shared element schema before scoring, with field-specific matching and normalization rules. Unparsable outputs are repaired when possible, and otherwise only affected fields are treated as empty.

D Evaluation Details

  • Layout matching requires category agreement and IoU of at least 0.5; reading order is scored over matched element identifiers, and OCR is normalized before edit-distance scoring.Table content uses the serialized HTML field when available, while missing applicable visual descriptions receive zero for that description metric.Source: D Evaluation Details · p. 20
  • Recoverable syntax errors are repaired deterministically, while remaining unparsable predictions leave only affected fields empty.This localizes syntax failure to fields it prevents from being evaluated and avoids credit for content that cannot be reliably associated with an element.Source: D Evaluation Details · p. 20

E Training-data Scaling

SARA’s structural advantage over full fine-tuning emerges at larger training fractions in this experiment. Lower input resolution reduces several structural metrics, while dense composite figures and tables expose alignment and decoding failures.

E Training-data Scaling

  • −0.0756 at 10%, +0.1790 at 50%, and +0.1853 at 100%: SARA’s F1 difference versus full fine-tuning grows with training fraction.At 50%, SARA improves F1, mIoU, and OCR while matching RO; at 100%, it improves all four structural metrics. METEOR favors SARA at every fraction, while BERTScore changes are mixed.Source: E Training-data Scaling · pp. 20–21, Table 13 · p. 21

F Resolution Sensitivity

  • At 25% of native crop resolution, F1 falls by 0.1600, mIoU by 0.0725, OCR by 0.1988, and RO by 0.0910.The remaining mIoU reflects overlap among surviving matches, not preservation of all element boundaries; the experiment does not compare robustness against full fine-tuning at matched resolutions.Source: F Resolution Sensitivity · p. 21, Table 14 · p. 21

G Failure cases.

  • Borderless composite figures can shift predicted figure boxes away from captions, while dense tables can truncate long outputs or produce minor irrelevant artifacts.Table-region localization may remain relatively accurate despite decoding problems.Source: G Failure cases · p. 21, Figure 5 · p. 22
Loading 2610.20335v1…