PaperlayerView exact revision on arXiv

Source-linked AI summary

MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm

Zhang Li, Yuliang Liu, Qiang Liu, Zhiyin Ma, Ziyang Zhang, Shuo Zhang, Biao Yang, Zidun Guo, Jiarui Zhang, Xinyu Wang, Xiang Bai

arXiv:2506.05218v2cs.CV

TL;DR

Document parsing systems face cumulative pipeline errors and inefficiencies from full-page end-to-end processing. MonkeyOCR addresses these issues with SRR, MonkeyDoc, and CPD, reporting state-of-the-art performance with faster inference and lightweight variants.

  • Problem

    Existing document parsing methods are limited by cumulative errors in multi-tool pipelines, lengthy-context inefficiencies in end-to-end models, and datasets focused on single tasks, languages, or document types.

  • Method

    MonkeyOCR uses SRR to separate structure detection, block-level content recognition, and reading-order relation prediction, trained with the 4.5-million-instance MonkeyDoc dataset and supported by CPD.

  • Results

    MonkeyOCR achieves state-of-the-art performance on OmniDocBench, including 7 of 10 subtasks, while running 2.09× faster than the fully end-to-end baseline.

  • Takeaways & Limitations

    The framework offers a unified document-parsing approach with efficient inference and lightweight models spanning 0.6B–1.2B parameters.

  • Takeaways & Limitations

    Layout-detection errors can propagate to recognition, while photographed documents, semantically complex layouts, and cross-page tables remain challenging.

Abstract

from arXiv · show

We introduce MonkeyOCR, a document parsing model that advances the state of the art by leveraging a Structure-Recognition-Relation (SRR) triplet paradigm. This design simplifies what would otherwise be a complex multi-tool pipeline and avoids the inefficiencies of processing full pages with giant end-to-end models. In SRR, document parsing is abstracted into three fundamental questions - ``Where is it?'' (structure), ``What is it?'' (recognition), and ``How is it organized?'' (relation) - corresponding to structure detection, content recognition, and relation prediction. To support this paradigm, we present MonkeyDoc, a comprehensive dataset with 4.5 million bilingual instances spanning over ten document types, which addresses the limitations of existing datasets that often focus on a single task, language, or document type. Leveraging the SRR paradigm and MonkeyDoc, we trained a 3B-parameter document foundation model. We further identify parameter redundancy in this model and propose contiguous parameter degradation (CPD), enabling the construction of models from 0.6B to 1.2B parameters that run faster with acceptable performance drop. MonkeyOCR achieves state-of-the-art performance, surpassing previous open-source and closed-source methods, including Gemini 2.5-Pro. Additionally, the model can be efficiently deployed for inference on a single RTX 3090 GPU. Code and models will be released at https://github.com/Yuliang-Liu/MonkeyOCR.

1. Introduction

Document parsing must handle complex multimodal documents, but existing pipeline and end-to-end approaches face complementary error and efficiency limitations. MonkeyOCR addresses this with SRR, MonkeyDoc, and models designed for accuracy and faster inference.

  • Document parsing converts multimodal document content into structured information for applications including offices, education, medical records, judicial documents, and historical preservation.
  • Pipeline toolchains divide parsing into specialized sequential subtasks, while giant end-to-end models process entire page images holistically.Pipeline modularity permits independent component optimization, whereas end-to-end models streamline the workflow.
  • Pipeline methods accumulate errors across detection, recognition, merging, and reading-order stages, while end-to-end models face performance and efficiency limitations on lengthy contexts.Imprecise formula detection can crop characters and produce erroneous recognition, such as extraneous superscripts.
  • MonkeyOCR uses SRR: structure detection identifies regions, block-level recognition processes them in parallel, and relation prediction determines reading order before integration.The design separates where elements are, what they contain, and how they are organized.
  • 4.5 million bilingual instances across five parsing tasks and more than ten document types comprise MonkeyDoc, addressing datasets limited by task, language, or document type.
  • MonkeyOCR combines a 3B foundation model with CPD, producing 0.6B–1.2B lightweight models and achieving stronger performance and faster inference than cited baselines.MonkeyOCR reportedly surpasses PPStruct-V3 by 4.1%, runs 2.09× faster than an end-to-end baseline, and its 1.2B model is 34% faster than the 3B model with a 1.5% performance drop.

2. Related Work

Related work organizes document parsing around pipeline methods, end-to-end models, and task-focused datasets. These approaches establish broad capabilities but leave room for more comprehensive parsing resources.

  • Pipeline-based approaches: Pipeline approaches decompose document parsing into sequential specialized models for layout, reading order, OCR, formulas, and tables.Docling is described as a linear PDF-processing pipeline that extracts page content through OCR, layout, and table components.
  • End-to-end models: End-to-end systems directly map document images to structured outputs, with examples spanning OCR-free document understanding, Markdown conversion, text spotting, extraction, and table recognition.
  • Document parsing datasets: Document-parsing datasets commonly target specific document types, languages, or subtasks such as layout detection and content recognition.Layout datasets provide annotations for structural-element positions across diverse document types.

3. MonkeyDoc Dataset

MonkeyDoc is built as a comprehensive bilingual dataset spanning the document parsing process and more than ten document types. Its generation pipeline combines existing annotations, synthesis, model labeling, filtering, and manual validation.

  • MonkeyDoc covers more than ten document types, supports Chinese and English, and spans the entire document parsing process.
  • Structure detection: Structure-detection data combines annotations from open-source datasets with additional Chinese pages, automatic pre-annotation, rule-based filtering, and manual correction.The process removes nested boxes and selects structurally rich samples to increase diversity.
  • Content recognition: 2.5 million document-element sub-images support content recognition, with model-based annotation and filtered samples from table and formula datasets.The construction also addresses Chinese-data scarcity by populating existing table structures with Chinese content.
  • Relation prediction: Relation-prediction data targets block-level reading order by linking tables and images to captions, removing low-quality samples, and manually annotating Chinese document types.

4. MonkeyOCR

MonkeyOCR processes documents through structure detection, block-level content recognition, and relation prediction, then integrates the outputs into a structured document. Its relation model uses category-aware positional features and greedy decoding, while experiments examine accuracy–speed trade-offs through layer pruning.

  • SRR pipeline: MonkeyOCR first detects document-element bounding boxes and categories, then crops regions for content recognition and predicts their reading order.The recognized outputs are integrated according to the predicted order to produce the final structured document.
  • Structure detection: The structure detector uses a DETR-based architecture with visual feature extraction, query selection, iterative decoder attention, and prediction heads.Its predictions include each element’s bounding box and category label.
  • Content recognition: Content recognition preserves cropped regions at original resolution, encodes them visually, and combines them with category-specific prompts in an LLM.Prompts are selected for block types such as text, tables, and formulas.
  • Relation prediction: The relation model combines bounding-box geometry with category-aware embeddings to produce reading-order logits through transformer layers.The model explicitly represents element categories because document types can have distinct semantic priorities for reading order.
  • Relation prediction: Greedy decoding assigns elements to their highest-scoring positions, resolves conflicts using logits, and repeats reassignment until a valid permutation is obtained.This converts the predicted logits into the document’s final reading order.
  • Efficiency analysis: Removing the first LLM layer causes a notable performance drop, whereas pruning other layers produces relatively minor degradation in sampled text, formula, and table recognition experiments.The study randomly selected 50 samples from each task and pruned one layer at a time.

5. Experiments

Experiments show that MonkeyOCR performs strongly across document-parsing tasks while preserving efficiency through SRR and CPD. Ablations support the contributions of SRR, MonkeyDoc, block-level reading-order data, category-aware embeddings, and contiguous parameter degradation.

  • Comparison on Different Tasks: MonkeyOCR achieves the best overall performance on Chinese and English OmniDocBench tasks, with state-of-the-art results on 7 of 10 subtasks.
  • Comparison on Different Tasks: 13.1% improvement in formula recognition over PPStruct-V3 is achieved by the 3B model, which also outperforms Gemini2.5-Pro.
  • SRR and MonkeyDoc Ablation: 28.6% average overall improvement is obtained from SRR in zero-shot parsing, including a 27.8% gain in text recognition.
  • SRR and MonkeyDoc Ablation: 4.2% and 14.8% decreases in text and formula recognition occur when additional detection models are introduced, supporting SRR’s mitigation of cumulative errors.
  • Inference Speed: 34% speedup with only a 1.5% performance drop is achieved by MonkeyOCR-1.2B versus MonkeyOCR-3B, while MonkeyOCR-3B runs 2.09× faster than the end-to-end baseline.
  • Reading Order Ablation: 5.6% better reading-order performance follows training on block-level data, with further gains from category information and minor effects on other tasks.
  • Pruning Ablation: 5.8% higher performance than training from scratch is achieved by CPD, which surpasses the other tested pruning strategies.
  • CPD Ablation: 12% table-recognition decline occurs when only four layers are retained, compared with a 3B model’s 1.5% overall drop when twelve layers remain.

6. Conclusion

MonkeyOCR unifies structural detection, content recognition, and relation prediction through SRR, simplifying document parsing while avoiding inefficient full-page processing. CPD further accelerates inference with minimal performance loss, and experiments report state-of-the-art OmniDocBench performance.

  • MonkeyOCR unifies structural detection, content recognition, and relation prediction into a streamlined SRR framework.
  • CPD accelerates inference by exploiting parameter redundancy while maintaining a minimal performance drop.
  • MonkeyOCR achieves state-of-the-art OmniDocBench performance and surpasses the closed-source Gemini2.5-Pro model.

A. MonkeyDoc Dataset

MonkeyDoc is designed as a broad bilingual resource for document parsing, spanning diverse tasks, document domains, and annotation sources. Its construction combines open-source filtering, synthesis, manual annotation, model labeling, and quality control.

  • Dataset Scope: MonkeyDoc covers layout detection, reading order, text, table, formula, and code recognition across more than ten document domains in Chinese and English.
  • Construction Strategy: The dataset-generation pipeline combines manual annotation, programmatic synthesis, and model-driven automatic labeling to address narrow task, language, and domain coverage.
  • Content Recognition: 2.5 million cropped document-element samples support text, formula, table, and code recognition data construction.
  • Content Recognition: 470k quality-filtered tables are obtained from PubTabNet after checks for structure, merged cells, alignment, encoding, and syntax.
  • Relation Prediction: 951k high-quality region-level reading-order samples result from correcting DocGenome annotations, filtering low-quality pages, and selecting diverse pages.
  • Relation Prediction: 154k manually annotated Chinese samples expand reading-order coverage across reports, papers, manuals, books, slides, newspapers, and other document types.

A.4. Comparison with Human Annotation.

Additional evaluations examine annotation quality, backbone robustness, benchmark performance, and behavior on non-Manhattan layouts. The reported visualizations include rectangular detection of non-rectangular text and successful reading-order handling in an artistic magazine.

  • Annotation Quality: Qwen2.5VL-72B was used for text and formula annotation, while Gemini 2.5-Pro handled tables after pilot quality checks found weaker table-recognition performance.
  • Backbone Comparison: MonkeyOCR remains robust across evaluated encoder and language-model backbone configurations, with larger backbones generally achieving better overall performance.
  • Benchmark Comparison: OmniDocBench v1.5 contains 1,355 document images with more balanced document and element-type distributions, and evaluates formula recognition using CDM.
  • Non-Manhattan Layouts: Non-rectangular text blocks are detected as rectangular regions, while image order in an artistic magazine does not affect correct textual reading order.
  • Non-Manhattan Layouts: When structure detection fails, the system performs end-to-end recognition on the full page.

E. More analysis of CPD

CPD is more training-efficient than knowledge distillation while preserving stronger performance under the reported setup.

  • CPD achieves approximately three times higher training efficiency than knowledge distillation because KD requires additional inference from the 3B teacher model.
  • 2.17%: CPD outperforms knowledge distillation under the same training setup.
  • CPD constructs the 1.2B model by selecting layers from Qwen2.5-3B and using MonkeyOCR-3B as the teacher for comparison with KD.

F. Limitations

MonkeyOCR simplifies pipeline processing but remains constrained by layout-detection errors and difficult multi-page or semantically complex document structures.

  • Pipeline methods still involve cumulative errors across layout detection, recognition, merging, and reading-order reconstruction despite the proposed simplification.
  • Layout-detection errors can propagate to subsequent recognition steps, limiting reliability when rectangular boxes are inaccurate for photographed documents.
  • Complex layouts requiring semantic understanding may remain insufficiently handled by the method.
  • Cross-page tables, paragraphs, hierarchical headings, and multi-page document parsing remain persistent challenges.

G. Comparison with other Methods

MonkeyOCR is compared with pipeline methods and giant end-to-end models, with the supplied passages emphasizing formula and table-recognition behavior.

  • Pipeline methods often produce unnecessary superscripts and incorrect symbols in inline formulas because inaccurate detection accumulates errors across processing stages.
  • MonkeyOCR-3B accurately reconstructs table structure and content despite having fewer parameters than Qwen2.5VL-7B and InternVL3.5-8B.
  • Qwen2.5VL-7B shows table-structure errors, while InternVL3.5-8B additionally makes symbol-recognition mistakes in the visual comparison.
Loading 2506.05218v2…