Source-linked AI summary
Dolphin-v2: Universal Document Parsing via Scalable Anchor Prompting
Hao Feng, Wei Shi, Ke Zhang, Xiang Fei, Lei Liao, Dingkang Yang, Yongkun Du, Xuecheng Wu, Jingqun Tang, Yang Liu, Hong Chen, Can Huang
TL;DR
Document parsing systems remain fragmented, and prior two-stage methods struggle with photographed or distorted documents. Dolphin-v2 addresses this with document-type-aware layout analysis and hybrid parsing, achieving +14.78 points on OmniDocBench and 91% error reduction on photographed documents. Its scope remains bounded by occasional document-type misclassification and unsupported specialized elements.
Problem
Existing document parsing systems have fragmented capabilities, while axis-aligned layout detection in two-stage methods performs poorly on photographed or geometrically distorted documents.
Method
Dolphin-v2 jointly classifies document type and analyzes layout, then parses photographed pages holistically and digital documents element-wise in parallel using layout anchors.
Results
+14.78 points on OmniDocBench and 91% error reduction on photographed documents were reported while maintaining efficient parallel processing.
Takeaways & Limitations
The framework combines robust photographed-document parsing with efficient digital-document extraction and broader element coverage, including metadata, formulas, and code blocks.
Takeaways & Limitations
Borderline photographed documents may be misclassified as digital, and chemical structures, charts, visualizations, and musical notations remain unsupported.
Abstract
from arXiv · showhide
Document parsing has garnered widespread attention as vision-language models (VLMs) advance OCR capabilities. However, the field remains fragmented across dozens of specialized models with varying strengths, forcing users to navigate complex model selection and limiting system scalability. Moreover, existing two-stage approaches depend on axis-aligned bounding boxes for layout detection, failing to handle distorted or photographed documents effectively. To this end, we present Dolphin-v2, a two-stage document image parsing model that substantially improves upon the original Dolphin. In the first stage, Dolphin-v2 jointly performs document type classification (digital-born versus photographed) alongside layout analysis. For digital-born documents, it conducts finer-grained element detection with reading order prediction. In the second stage, we employ a hybrid parsing strategy: photographed documents are parsed holistically as complete pages to handle geometric distortions, while digital-born documents undergo element-wise parallel parsing guided by the detected layout anchors, enabling efficient content extraction. Compared with the original Dolphin, Dolphin-v2 introduces several crucial enhancements: (1) robust parsing of photographed documents via holistic page-level understanding, (2) finer-grained element detection (21 categories) with semantic attribute extraction such as author information and document metadata, and (3) code block recognition with indentation preservation, which existing systems typically lack. Comprehensive evaluations are conducted on DocPTBench, OmniDocBench, and our self-constructed RealDoc-160 benchmark. The results demonstrate substantial improvements: +14.78 points overall on the challenging OmniDocBench and 91% error reduction on photographed documents, while maintaining efficient inference through parallel processing.
1 INTRODUCTION
Dolphin-v2 extends two-stage document parsing to digital-born and photographed documents through document-type-aware processing, finer-grained layout analysis, and hybrid content parsing. It reports substantial gains on OmniDocBench and photographed-document evaluation while preserving efficient parallel processing.
- Hybrid parsing: Photographed documents are parsed holistically, while digital documents receive parallel element-wise parsing guided by layout anchors.This hybrid strategy addresses geometric distortions in photographed pages while retaining efficient decoding for digital documents.
- Evaluation: +14.78 points improvement on OmniDocBench and 91% error reduction on photographed documents demonstrate the reported performance gains.The results cover diverse evaluation dimensions and the RealDoc-160 photographed-document benchmark.
- Architecture: Dolphin-v2 handles digital-born and photographed documents through a document-type-aware two-stage architecture with hybrid parsing.The first stage combines document-type classification with layout analysis, while the second stage selects holistic or element-wise parsing based on document type.
- Layout analysis: 21 element categories, absolute coordinates, reading-order prediction, and semantic attributes expand layout analysis beyond the original system.The enhancements target precise localization and metadata such as author information and publication details.
- Specialized parsing: The paper adds dedicated formula and code-block modules, with code parsing preserving indentation structure.Formula outputs use precise LaTeX representations, while indentation supports programming-language structure.
2 RELATED WORK
Document parsing research spans modular pipelines and unified vision-language models, with each paradigm offering distinct benefits and limitations. Dolphin-v2 builds on localized element parsing while addressing limited coverage and missing code support.
- Integration-based Document Parsing: Integration-based systems cascade specialized models for layout detection, text recognition, tables, and other document elements.
- Integration-based Document Parsing: These modular pipelines provide domain-specific extraction but increase complexity, coordination difficulty, and limitations in capturing intricate layouts.
- End-to-End Document Parsing with VLMs: Unified VLM approaches simplify document parsing architecture by using autoregressive language modeling instead of coordinating multiple specialized components.
- End-to-End Document Parsing with VLMs: Existing element-wise methods have limited category coverage, lack semantic attribute extraction, and generally do not support indentation-preserving code recognition.
3 METHODOLOGY
Dolphin-v2 uses a two-stage vision-language pipeline that classifies document type and analyzes layout before applying type-specific content parsing. It combines holistic processing for photographed pages with parallel element parsing for digital documents.
- Overview: Dolphin-v2 jointly classifies pages as digital or photographed and performs layout analysis before downstream parsing.
- Joint Classification and Layout Analysis: Digital-document layout modeling expands element detection from 14 to 21 categories and adds semantic attributes such as author and publication information.
- Joint Classification and Layout Analysis: Absolute pixel coordinates replace normalized coordinates for bounding-box prediction, improving spatial localization precision.
- Hybrid Content Parsing: For photographed documents, the model parses the complete page holistically to accommodate distortions, perspective changes, and irregular layouts.
- Hybrid Content Parsing: For digital documents, Dolphin-v2 detects elements with reading order, crops them, and parses them in parallel using type-specific prompts.
- Type-specific Parsing: Dedicated prompts generate LaTeX for formulas, preserve indentation in code blocks, produce HTML for tables, and recognize paragraphs efficiently.
4 DATASETS
Dolphin-v2 is supported by synthesized training data and complementary benchmarks covering photographed documents, code, catalogs, and diverse parsing capabilities. The evaluation resources emphasize robustness under realistic capture conditions and fine-grained structural analysis.
- Training Data: The authors synthesize additional training images for photographed documents, code, and catalogs with corresponding OCR annotations.
- Training Data: 200K photographed-document images model realistic creases, folds, and perspective distortions through physics-based deformation rendering.
- Training Data: 200K code images cover C++, Python, Go, and JavaScript while providing annotations for code boundaries, content, and indentation.
- Training Data: Catalog data includes single- and double-column layouts, hierarchical structures, varied entry counts, and randomized formatting.
- Evaluation Benchmarks: The evaluation uses OmniDocBench, RealDoc-160, and DocPTBench to assess diverse document parsing capabilities and photographed-document robustness.
- Evaluation Benchmarks: RealDoc-160 contains 160 photographed English and Chinese pages captured by mobile phones under varied lighting, viewpoints, and paper deformations.
- Evaluation Benchmarks: DocPTBench contains more than 1,300 high-resolution photographed documents with human-verified annotations and realistic capture challenges.
5 EXPERIMENTS
Experiments evaluate Dolphin-v2 across standard, photographed-document, qualitative, and ablation settings. Results show strong parsing quality, robustness to distortions, code-structure preservation, and benefits from document-type and formula-specific handling.
- Qualitative Analysis: Dolphin-v2 handles photographed documents with perspective transformations, wrinkles, and nonuniform illumination while preserving text recognition and structure.Its holistic strategy processes the entire page as unified context rather than isolated elements.
- Qualitative Analysis: Dolphin-v2 preserves code indentation and hierarchical structure through dedicated code parsing and explicit code-block layout categories.The specialized prompt guides attention to whitespace patterns and indentation levels.
- Qualitative Analysis: Dolphin-v2 decomposes complex pages into layout anchors for parallel content extraction, supporting formulas and bibliography entries in dense academic documents.The first-stage layout predictions reduce the structural complexity presented to second-stage parsing.
- Quantitative Results: 89.45 overall score on OmniDocBench exceeds the original Dolphin’s 74.67 by +14.78 points.The 3B-parameter model also reports 0.054 Edit distance for text recognition and reading-order prediction and 90.48 TEDS-S for table parsing.
- Quantitative Results: 0.0392 Edit Distance on RealDoc-160 is the best average performance reported for photographed-document parsing.Dolphin-v2 scores 0.0046 on English documents and 0.0737 on Chinese documents.
- Quantitative Results: 30.8 and 37.3 Edit distances on DocPTBench improve over the original Dolphin’s 57.5 and 71.5 for English and Chinese photographed documents.Dolphin-v2 achieves the best overall performance among specialized VLMs, while Gemini2.5-Pro remains strongest overall.
- Ablation Studies: 377% degradation in average Edit Distance occurs without document-type classification, increasing from 0.0392 to 0.1871 on RealDoc-160.The ablation attributes this gap to the inability of one parsing strategy to optimize both digital and photographed documents.
- Ablation Studies: +3.38 points in formula parsing accuracy results from separate formula detection and specialized LaTeX generation prompts.The reported CDM improves from 83.34 without dedicated handling to 86.72 with it.
6 LIMITATION DISCUSSIONS
The paper identifies document-type misclassification and incomplete element coverage as limitations. Borderline photographed documents may receive inappropriate element-wise parsing, while several specialized element types remain unsupported.
- Document Type Classification Errors: Mildly distorted photographed documents can be misclassified as digital, causing inappropriate element-wise parsing instead of holistic page parsing.The issue is most likely near perpendicular capture angles with minimal wrinkles or lighting variation, and errors may propagate to the second stage.
- Element Type Coverage: Dolphin-v2 currently supports text paragraphs, formulas, tables, and code blocks but does not yet cover chemical structures, charts, data visualizations, or musical notation.The paper presents these specialized element types as targets for future extension.
- Element Type Coverage: The anchor-based framework can add new element types through category labels and type-specific prompts without fundamental architectural changes.This scalability is presented as a basis for extending coverage to emerging document types and user requirements.
7 CONCLUSION
Dolphin-v2 combines document-type classification, layout analysis, and hybrid parsing in a two-stage framework. The paper reports improved accuracy and computational efficiency across digital and photographed documents, while outlining broader applications and future extensions.
- Conclusion: Dolphin-v2 jointly classifies document type and analyzes layout before parsing photographed pages holistically and digital documents element-wise in parallel.This design targets both accuracy and computational efficiency across document types.
- Conclusion: The model adds 21-category element detection, reading-order prediction, semantic attribute extraction, absolute coordinates, and dedicated formula and code parsing.Code parsing includes indentation preservation.
- Conclusion: +14.78 points on OmniDocBench and 91% error reduction on photographed documents summarize the reported gains over the original Dolphin.The conclusion characterizes the architecture as bridging specialized and general document parsing capabilities.
- Broad Impact: The paper identifies document digitization, accessibility, and enterprise knowledge management as potential application areas.It specifically highlights photographed documents with distortions as valuable in resource-limited settings.
- Future Work: Future work includes stronger borderline document classification, expanded element coverage, cross-page context modeling, and downstream LLM integration.Proposed additions include chemical structures, complex charts, musical notations, and multi-page consistency mechanisms.