PaperlayerView exact revision on arXiv

Source-linked AI summary

Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting

Hao Feng, Shu Wei, Xiang Fei, Wei Shi, Yingdong Han, Lei Liao, Jinghui Lu, Binghong Wu, Qi Liu, Chunhui Lin, Jingqun Tang, Hao Liu, Can Huang

arXiv:2505.14059v1cs.CV

TL;DR

Document image parsing must recover structure from pages containing intertwined elements, while existing approaches face integration, efficiency, and layout-preservation challenges. Dolphin uses two-stage heterogeneous anchor prompting to analyze layouts before parsing elements in parallel, and achieves state-of-the-art performance with efficiency advantages. Its scope remains limited for vertical text, broader multilingual documents, and handwriting recognition.

  • Problem

    Document image parsing must extract intertwined text, figures, formulas, and tables, while existing methods face integration overhead, efficiency bottlenecks, and layout-structure degradation.

  • Method

    Dolphin first generates structured layout elements in reading order, then uses them as anchors with type-specific prompts for parallel element-level content parsing.

  • Results

    Dolphin achieves state-of-the-art performance across diverse page-level and element-level parsing tasks while providing running-efficiency advantages from its lightweight architecture and parallel parsing mechanism.

  • Takeaways & Limitations

    The two-stage design balances efficiency and accuracy while handling complex documents with interleaved tables, formulas, and rich formatting in Chinese and English.

  • Takeaways & Limitations

    Dolphin has limited capability for vertical text, needs broader multilingual capacity, and requires further enhancement for handwriting recognition.

Abstract

from arXiv · show

Document image parsing is challenging due to its complexly intertwined elements such as text paragraphs, figures, formulas, and tables. Current approaches either assemble specialized expert models or directly generate page-level content autoregressively, facing integration overhead, efficiency bottlenecks, and layout structure degradation despite their decent performance. To address these limitations, we present \textit{Dolphin} (\textit{\textbf{Do}cument Image \textbf{P}arsing via \textbf{H}eterogeneous Anchor Prompt\textbf{in}g}), a novel multimodal document image parsing model following an analyze-then-parse paradigm. In the first stage, Dolphin generates a sequence of layout elements in reading order. These heterogeneous elements, serving as anchors and coupled with task-specific prompts, are fed back to Dolphin for parallel content parsing in the second stage. To train Dolphin, we construct a large-scale dataset of over 30 million samples, covering multi-granularity parsing tasks. Through comprehensive evaluations on both prevalent benchmarks and self-constructed ones, Dolphin achieves state-of-the-art performance across diverse page-level and element-level settings, while ensuring superior efficiency through its lightweight architecture and parallel parsing mechanism. The code and pre-trained models are publicly available at https://github.com/ByteDance/Dolphin

1 Introduction

Document image parsing must organize intertwined text, figures, formulas, and tables into machine-readable structure, but existing approaches face integration, efficiency, and layout-preservation challenges. Dolphin addresses these issues with a two-stage analyze-then-parse design and reports broad benchmark strength with efficient parallel parsing.

  • Motivation: Document image parsing extracts structured content from images containing intertwined text paragraphs, figures, tables, and formulas for downstream content analysis.Its importance has increased with the growth of digital documents across academic, business, and technical domains.
  • Challenges: Existing solutions either integrate specialized models or use end-to-end vision-language generation, creating coordination, layout-structure, or efficiency challenges.The cited limitations are especially relevant to long documents with complex layouts.
  • Dolphin: Dolphin follows an analyze-then-parse paradigm that first generates page-level elements in reading order and then uses them as anchors for element-specific parsing.The first stage preserves relationships such as figure-caption pairs, table-caption associations, and section title-paragraph hierarchies.
  • Training: Dolphin is trained on 30 million samples spanning page-level documents and element-level blocks.Element-decoupled parsing also makes isolated element images and annotations more feasible to collect than full pages with diverse elements.
  • Results: Dolphin achieves state-of-the-art performance across diverse page-level and element-level tasks while providing running-efficiency advantages from its lightweight architecture and parallel parsing mechanism.The evaluations include prevalent and self-constructed benchmarks.

2 Related Work

Document image parsing research spans integrated specialist pipelines and end-to-end vision-language models. These approaches range from general-purpose models to document-specific systems, while Dolphin is presented within the broader end-to-end parsing landscape.

  • Overview: Document image parsing extracts structured content from rendered document images without requiring source file formats or parsing libraries.Existing solutions are categorized into integration-based and end-to-end approaches.
  • End-to-end Methods: End-to-end document parsing with vision-language models directly generates structured results through autoregressive decoding.Recent vision-language advances have established this as a distinct research paradigm.
  • Integration-based Methods: Integration-based methods assemble multiple specialized models, typically combining layout detection with dedicated recognizers for tables, formulas, and other elements.Commercial and academic systems such as Mathpix, TextIn, and MinerU follow this multistage paradigm.
  • General VLMs: General-purpose vision-language models demonstrate document-understanding results without task-specific training by leveraging large-scale pre-training.Examples include GPT-4V, Claude, Gemini, QwenVL, MiniCPM, InternVL, DeepSeek-VL2, and Step-1V.
  • Expert VLMs: Expert vision-language models are specifically designed and trained for document parsing or understanding tasks, including Nougat, GOT, Donut, LayoutLM, and related systems.Nougat converts documents into markup language, while GOT provides a unified model for varied document elements.

3 Approach

Dolphin uses a shared encoder-decoder vision-language model in two stages: page-level layout analysis produces reading-order anchors, and type-specific prompts guide parallel element parsing. This design operates across page and element granularities while supporting structured layout relationships.

  • 3.1 Overview: Dolphin uses a shared encoder-decoder transformer for two stages that operate at different input granularities with distinct prompting strategies.Stage 1 analyzes the page, while Stage 2 parses individual elements in parallel.
  • 3.2 Page-level Layout Analysis: The page-level stage encodes the resized and padded document image with a Swin Transformer into visual embeddings.The hierarchical encoder captures global layout patterns and local textual details while preserving the image aspect ratio.
  • 3.2 Page-level Layout Analysis: The decoder uses a layout-analysis prompt to generate document elements sequentially in natural reading order.Each layout element specifies its type and bounding box, forming anchors for later parsing.
  • Evaluation Setup: Table 1 compares page-level parsing on plain documents containing only text and complex documents mixing tables, formulas, and figures.Its arrows indicate whether higher or lower values are better; starred results are reported by GOT, bold marks the best, and underlining marks the second-best.
  • 3.2 Page-level Layout Analysis: The generated layout sequence preserves structural relationships including figure-caption pairs, table-caption associations, and section title-paragraph hierarchies.These relationships are retained in the structured sequence used by the second stage.
  • 3.3 Element-level Content Parsing: The element-level stage crops each identified region, encodes local views in parallel, and applies type-specific prompts to generate element content.Tables use dedicated prompts for HTML, while formulas share the paragraph prompt because they occur inline and in display mode within paragraph contexts.

4 Dataset

Dolphin is trained and evaluated with multi-granularity document data spanning page-level layouts and element-level text, formulas, and tables. The dataset combines synthetic and public sources to support both layout-aware parsing and fine-grained recognition.

  • Dataset Construction: Over 30 million samples cover page-level documents and element-level components for different parsing objectives.The paper states that page-level documents are also decomposed into individual elements for element-specific training.
  • Dataset Construction: The training data includes LaTeX, Markdown, mixed documents, HTML-rendered pages, tables, and formula images.These sources provide annotations for hierarchy, spatial locations, table structure, and formula content at multiple granularities.
  • Element Data: Table sources include 568K PubTabNet tables with HTML annotations and 1M PubTab1M tables with finer-grained structure annotations.These datasets are used specifically for table parsing.
  • Evaluation Data: Evaluation spans page-level benchmarks for pure-text and complex documents and element-level tests for text paragraphs, formulas, and tables.The page-level settings include Fox-Page and Dolphin-Page, while element-level evaluation uses public text, formula, and table test sets.
  • Evaluation Data: Dolphin-Page contains 210 bilingual pages, including 99 challenging samples with interleaved tables, formulas, and figures across single- and multi-column layouts.All pages are manually annotated with precise transcriptions in natural reading order.

5 Experiment

Experiments evaluate Dolphin on full-page parsing, element recognition, and component ablations. Dolphin reports strong accuracy with a 322M-parameter model, while parallel decoding improves efficiency without reducing parsing accuracy.

  • Page-level Parsing: Dolphin achieves edit distances of 0.0114 and 0.0131 on Fox-Page English and Chinese, respectively, and 0.1283 on Dolphin-Page.The reported values outperform the listed specialized and general VLM baselines, including on mixed-element documents.
  • Page-level Parsing: Dolphin’s 322M-parameter architecture reaches 0.1729 FPS, nearly 2× the 0.0944 FPS of Mathpix.The comparison is reported for page-level parsing with the parallel parsing design.
  • Element-level Parsing: Element-level experiments report competitive performance for text paragraphs, formulas across SPE, SCE, and CPE, and tables on PubTabNet and PubTab1M.The paper summarizes consistent results across these fundamental recognition tasks.
  • Ablation Studies: Parallel decoding provides a 1.8× speedup, from 0.0971 to 0.1729 FPS, while maintaining the same parsing accuracy.The speedup is bounded by per-element preprocessing and a maximum batch size of 16 elements under GPU memory constraints.
  • Ablation Studies: Type-specific prompts improve Dolphin-Page edit distance from 0.1613 with a generic prompt to 0.1283.The generic prompt misidentifies a table as a LaTeX formula, whereas type-specific prompting renders it as HTML.
  • Ablation Studies: Element cropping outperforms box queries because it gives the model a focused view instead of combining location understanding with content recognition.The paper frames this as a “what you see is what you get” design choice.

6 Conclusion

Dolphin uses an analyze-then-parse paradigm that generates structured layout elements in reading order before parallel element parsing through heterogeneous anchor prompting. Experiments show strong page-level and element-level performance, including complex documents with interleaved tables, formulas, and rich formatting.

  • Dolphin first generates structured layout elements in reading order, then performs parallel element parsing through heterogeneous anchor prompting.This two-stage design uses a lightweight architecture to balance efficiency and accuracy.
  • Dolphin demonstrates strong performance on both page-level and element-level document parsing tasks.The reported strengths include complex documents with interleaved tables, formulas, and rich formatting in Chinese and English.

Limitations

Dolphin has limited support for vertical text and broader multilingual document parsing, while further optimization is needed for parallel fine-grained processing and handwriting recognition.

  • Dolphin primarily supports standard horizontal text layouts and has limited capability for vertical text such as ancient manuscripts.
  • Dolphin handles Chinese and English effectively, but its multilingual capacity still needs expansion.The supplementary materials include some emergent multilingual document parsing cases.
  • Further efficiency gains could come from parallel processing of text lines and table cells.
  • Handwriting recognition capabilities require further enhancement.

A Qualitative Results

Qualitative evaluations show Dolphin handling diverse document layouts, languages, element types, and text-rich everyday scenarios while preserving structural and textual content. Element-level cases further demonstrate formula recognition across formats and table parsing at large scale.

  • Page-level: Dolphin handles textbook pages with dense formulas, triple-column English papers, and double-column Chinese papers with tables.These cases span different languages, layouts, and element types while maintaining high parsing quality.
  • Page-level: Dolphin captures structural layout and textual content in mobile screenshots, shopping receipts, and webpage captures.
  • Element-level: Dolphin accurately parses inline, single-line block, and multi-line block formulas with varying complexity and layout formats.
  • Element-level: Dolphin successfully parses a table containing hundreds of cells with precise content recognition and layout preservation.

B Element Design

Dolphin supports diverse document elements through heterogeneous prompting and separates page-level layout analysis from element-level parsing. Its design uses specialized handling for tables while treating other elements as text paragraphs.

  • Element Types: Dolphin supports 15 element types covering common document structures from headers to specialized content blocks.
  • Element Design: Formulas are not independent Stage 1 elements, allowing Stage 2 recognition to use surrounding textual context.The design reflects formulas’ semantic connections with nearby text.
  • Heterogeneous Anchor Prompting: Heterogeneous prompting supports page layout analysis, paragraph parsing, table parsing, text spotting, text-box queries, and formula recognition.The latter two prompts provide flexible text recognition capabilities, while paragraph prompting can serve formula recognition.
  • Heterogeneous Anchor Prompting: Stage 2 uses a dedicated prompt for structured HTML table parsing and a unified prompt for all other elements as text paragraphs.This dichotomy distinguishes structured HTML from plain text and remains robust to element misclassification errors.

C Training Details

Dolphin’s training combines dynamic instruction-task selection, synthetic document data, and initialization from Donut to support diverse document-parsing prompts. Training uses cross-entropy loss over predicted and ground-truth token distributions.

  • Instruction Tuning: Dolphin uses five task types selected dynamically for instruction-based training, with prompts organized for document-parsing tasks.The supplied passages identify dynamic task selection and a prompt taxonomy but do not enumerate the five task names.
  • Training Data: Synthetic training data are generated from HTML, LaTeX, and Markdown documents with rendered images and paragraph-level annotations.Figure 7 presents representative examples for the three source formats.
  • Instruction Tuning: Dolphin randomly selects an applicable task for each training sample according to its available annotations and constructs question-answer pairs.A page with paragraph-level boxes and content annotations can support element-level text paragraph parsing and page-level box query parsing.
  • Model Initialization: Dolphin is initialized from pretrained Donut weights and instruction-tuned to analyze layout, reading order, paragraphs, tables, and formulas.Donut lacks instruction-following abilities, which instruction tuning extends for diverse prompts.
  • Training Loss: Training optimizes cross-entropy loss between predicted token distributions and ground-truth token sequences.This follows standard practice for autoregressive language models.

D Synthetic Data Examples

The examples illustrate synthetic document data and Dolphin’s staged parsing outputs across page layouts, text-rich images, formulas, and large tables. Visualizations connect layout analysis and reading order with rendered or element-specific parsing results.

  • Synthetic Data Examples: Synthetic examples pair rendered HTML, LaTeX, and Markdown documents with paragraph-level annotations.The annotations are visualized as colored regions beneath the rendered document examples.
  • Page-Level Parsing: Page-level examples show Stage 1 layout boundaries and reading order followed by Stage 2 element-specific outputs and a final Markdown rendering.The visualization presents input, intermediate layout analysis, element parsing, and final rendering across its panels.
  • Page-Level Parsing: Text-rich examples include mobile phone screenshots, shopping receipts, and webpage captures processed through layout analysis and downstream parsing.The outputs include final Markdown renderings for one row and element-specific parsing results for others.
  • Formula Parsing: Formula visualizations cover inline, single-line block, and multi-line block formulas, showing input images, LaTeX outputs, and rendered formulas.The figure presents the complete parsing pipeline for each formula type.
  • Table Parsing: A large-scale table example contains hundreds of cells and is paired with a rendered HTML table from Dolphin’s parsing result.The visualization focuses on structured table parsing at large scale.
Loading 2505.14059v1…