PaperlayerView exact revision on arXiv

Source-linked AI summary

MonkeyOCRv2: A Visual-Text Foundation Model for Document AI

Yuliang Liu, Zhang Li, Ziyang Zhang, Shuo Zhang, Qiang Liu, Jiajun Song, Zidun Guo, Xinhan Wang, Handong Zheng, Yang Liu, Dongliang Luo, Zhiyin Ma, Jiarui Zhang, Xiang Bai

arXiv:2607.11562v1cs.CV

TL;DR

Document images demand character-level visual perception that natural-image encoders do not provide. MonkeyOCRv2 addresses this gap with document-oriented visual pretraining and consistently improves five analysis tasks while enabling a compact parser with open-source state-of-the-art performance on MDPBench.

  • Problem

    Natural-image visual encoders lack the dense, fine-grained textual supervision and character-level perception needed for multilingual document understanding.

  • Method

    MonkeyOCRv2 jointly pretrains on image-to-text generation and pixel-level document reconstruction using the 113-million-sample, 17-language MonkeyDoc v2 corpus.

  • Results

    2.8% absolute improvement over the previous best 3B dots.mocr on MDPBench accompanies consistent gains across five document-analysis tasks and eight document-understanding benchmarks.

  • Takeaways & Limitations

    Document-oriented visual pretraining can serve as a foundation for document intelligence across diverse systems and tasks.

  • Takeaways & Limitations

    The frozen, minimally fine-tuned parser trails the strongest task-specific systems on saturated benchmarks such as OmniDocBench and favors accuracy over inference speed.

Abstract

from arXiv · show

Mainstream visual encoders are pretrained on natural images and cannot be effectively applied to document images without document-oriented adaptation, as dense text and fine-grained character strokes demand character-level visual perception. We present MonkeyOCRv2, a visual-text pretrained model for document AI. First, we construct MonkeyDoc v2, to our knowledge the largest document-image pretraining corpus, comprising 113 million images spanning 17 languages. Second, we propose a pretraining strategy that jointly learns image-to-text generation and pixel-level document reconstruction: the former aligns visual representations with textual content, while the latter preserves character strokes and layout details. Extensive experiments are conducted on five representative document analysis tasks, including text recognition, formula recognition, text detection, document tampering detection, and overlapping text segmentation. Replacing the original encoders with MonkeyOCRv2 consistently improves performance across all five tasks. Finally, we validate its effectiveness as the vision encoder of multimodal large language models on the more challenging tasks of document parsing and document understanding. Kept frozen and paired with a lightweight language model, it yields a 0.7B document parsing model that sets a new open-source state-of-the-art on MDPBench, a recent benchmark spanning digital-born and photographed documents across 17 languages, surpassing the previous best 3B dots.mocr by 2.8% absolute with a vision encoder roughly 11$\times$ smaller. The frozen encoder also powers a document understanding model that outperforms counterparts built on CLIP, DINO, and SAM across eight benchmarks under identical training settings. These results suggest that document-oriented visual pretraining can serve as a foundation for document intelligence in its own right.

1 Introduction

MonkeyOCRv2 addresses the mismatch between natural-image visual encoders and document images through document-oriented pretraining that preserves character-level and layout details. Built on the 113-million-image, 17-language MonkeyDoc v2 corpus, it transfers across document analysis, parsing, and understanding tasks.

  • Pretraining data: 113 million images across 17 languages comprise MonkeyDoc v2, described as the largest document-oriented visual-text pretraining dataset.The corpus covers diverse document types, including printed papers, scanned books, handwritten notes, newspapers, magazines, financial reports, and rendered documents.
  • Pretraining strategy: MonkeyOCRv2 jointly learns image-to-text generation and pixel-level document reconstruction to align visual features with text while preserving character strokes, glyph shapes, and layout structures.Generation provides direct, dense textual supervision, while reconstruction retains fine-grained visual evidence when linguistic context is weak or unavailable.
  • Document analysis transfer: 5.4% average absolute improvement in text recognition follows replacement of original visual encoders across challenging English, Chinese, and occluded-text benchmarks.The paper evaluates transfer on text recognition, formula recognition, text detection, document tampering detection, and overlapping text segmentation, reporting consistent gains across all five tasks.
  • Document parsing: 2.8% improvement over 3B dots.mocr is achieved on MDPBench by MonkeyOCRv2-Parsing while using a vision encoder roughly 11× smaller.The model uses a lightweight 0.1B visual encoder and achieves state-of-the-art performance among open-source models on the challenging multilingual benchmark.
  • Document understanding: MonkeyOCRv2 also outperforms mainstream pretrained encoders across eight document understanding benchmarks under the paper's stated transfer setting.The encoder is kept frozen and paired with a language model for document understanding.

2 Related Work

Related work progresses from general visual pretraining and OCR-specific representation learning toward multimodal document parsing and understanding. MonkeyOCRv2 addresses reproducibility and document-specific representation gaps through a standalone encoder, reconstruction-augmented pretraining, and evaluation across seven document analysis tasks.

  • General visual pretraining: Large-scale visual pretraining evolved from supervised ImageNet initialization toward self-supervised and vision-language paradigms for transferable representations.The supplied passage identifies supervised ImageNet pretraining as an early approach and notes later advances from self-supervised and vision-language pretraining.
  • General visual pretraining: Unified scalable methods combine contrastive, generative, and multi-teacher distillation objectives to improve semantic modeling and integrate diverse visual knowledge.OpenVision combines contrastive learning with generative supervision, OpenVision 2 uses generative image-to-text pretraining, and RADIO uses multi-teacher distillation.
  • Document-oriented pretraining: OCR-specific pretraining extends ImageNet-based document models with contrastive learning, masked image modeling, and character-aware vision-language approaches.DiG combines contrastive learning with masked image modeling and improves text recognition, text segmentation, and text image super-resolution.
  • Document AI backbones: Document AI has expanded toward parsing and understanding, but many systems still use general-purpose visual backbones or encoders initialized from multimodal language models.Examples include systems using SigLIP 2, OpenCLIP ViT-bigG, SAM, and encoders initialized from Keye-VL or Qwen2-VL.
  • Limitations of prior work: Massive non-public pretraining data in existing backbones limits reproducibility and accessibility for the research community.The passage specifically identifies this limitation for backbones used by recent document parsing models.
  • MonkeyOCRv2’s distinction: MonkeyOCRv2 pretrains a standalone encoder from scratch, adds reconstruction to generation, and evaluates backbone substitution across seven document analysis tasks.The tasks span recognition, detection, segmentation, tampering localization, parsing, and understanding; reconstruction is intended to preserve local visual evidence.

3 MonkeyOCRv2

MonkeyOCRv2 is pretrained on the multilingual MonkeyDoc v2 corpus using complementary text-generation and image-reconstruction objectives. The resulting encoder family spans three backbone variants and is designed to preserve both textual content and fine-grained document structure.

  • MonkeyDoc v2 corpus: MonkeyDoc v2 contains 113 million document samples spanning 17 languages, including 8M page-level images and 105M cropped document elements.Simplified and Traditional Chinese are counted as separate language entries.
  • Data construction: Multi-expert transcription selects the prediction with the highest average pairwise agreement, reducing model-specific errors in labels for cropped document elements.The pipeline first detects and crops text blocks, tables, formulas, and other regions, then independently transcribes each crop with complementary recognition models.
  • Data construction: Synthetic multilingual OCR data renders sampled corpus text and rare-character combinations with diverse fonts, styles, and resolutions, while filtering removes samples with incomplete layout annotations or residual text.The synthesis covers 17 languages and the filtering targets challenging sources such as newspapers and handwritten notes.
  • Pre-training objectives: Text generation aligns visual tokens with depicted text, whereas image reconstruction preserves character strokes, glyph structures, layout information, and other fine-grained visual evidence.The default reconstruction objective is mean squared error (MSE), and the overall loss combines text generation with image reconstruction using λ to balance them.
  • Architecture: MonkeyOCRv2 is trained as three from-scratch encoder variants: MonkeyOCRv2-S with 28M parameters, MonkeyOCRv2-B with 113M, and MonkeyOCRv2-AS with 21M.All variants use the same MonkeyDoc v2 corpus and dual-objective recipe; ViTAEv2-Small is selected for tasks benefiting from multi-scale inductive bias.

4 Evaluation on Downstream Tasks

MonkeyOCRv2 consistently improves downstream document-analysis performance across recognition, detection, tampering localization, parsing, and understanding tasks. Its gains include strong cross-domain recognition, robust detector compatibility, and competitive performance from frozen vision-language configurations.

  • Text recognition: 16.0% absolute average accuracy gain for CRNN and 3.3% for PARSeq on English Union14M, with PARSeq reaching 87.6% accuracy versus SVTRv2’s 86.1%.On the multi-scene Chinese benchmark, gains are 5.4% for CRNN and 1.3% for PARSeq.
  • Formula recognition: 0.9% CDM and 3.9% ExpRate absolute gains on OmniDocBench 1.6 show stronger generalization to real-world document formulas.MonkeyOCRv2 also surpasses the larger 325M UniMERNet-B, indicating that document-oriented visual features can compensate for reduced model capacity.
  • Text detection: F-measure improves across all four text-detection datasets and detector architectures, outperforming both ImageNet-pretrained and text-specific oCLIP encoders.In DBNet on ICDAR 2015, MonkeyOCRv2 raises F-measure from 85.0 to 88.5 and surpasses oCLIP by 1.1%.
  • Document tampering detection: 87.4% IoU and 93.3% F1-score on DocTamper-Test include an 11.3% absolute F1 gain over the baseline.Replacing FFDN’s visual encoder improves performance across all three test sets and achieves state-of-the-art overall results.
  • Document parsing: A frozen MonkeyOCRv2 encoder paired with a lightweight language model forms the 0.7B MonkeyOCRv2-Parsing model evaluated on MDPBench across digital-born and photographed documents in 17 languages.Encoder-level attribution relies on the controlled comparison in Tab. 8 rather than the system-level comparison in Tab. 7, whose differing components prevent isolating the visual encoder’s contribution.
  • Document understanding: Under identical training conditions with the same Qwen3-1.7B LLM, MonkeyOCRv2-backed models are evaluated against other backbones on eight document VQA benchmarks.The controlled setting keeps training data, optimization schedules, and decoding procedures identical while preserving each encoder’s native input configuration.

5 Discussion

The discussion shows that pixel-level reconstruction strengthens document parsing when linguistic context is unavailable, improving visual robustness and faithfulness. Qualitative comparisons further indicate that MonkeyOCRv2 preserves character-level details more effectively than general-purpose vision models.

  • Visual robustness: Existing VLMs lose substantial scrambled-text accuracy as resolution decreases, with drops of 38.5% for PaddleOCR-VL-1.5, 56.8% for dots.mocr, and 51.8% for Qwen2.5-VL-3B.The semantic–scrambled accuracy gap also widens as visual input quality degrades.
  • Visual robustness: With reconstruction, scrambled-text accuracy falls 27.6% from 1288 to 448 resolution, versus a 44.3% reduction without reconstruction.At the lowest resolution, accuracy remains 72.1% with reconstruction versus 55.4% without it.
  • Faithfulness: MonkeyOCRv2-B-Parsing achieves the best CHAOS-Bench performance, exceeding HunyuanOCR-1.5 by 3.7% and PaddleOCR-VL-1.6 by 11.9%.MonkeyOCRv2-S-Parsing also surpasses HunyuanOCR-1.5 by 0.5%.
  • Qualitative parsing: MonkeyOCRv2-Parsing produces more accurate results than competing models on both a two-column Arabic document and a photographed Chinese instruction manual.Competing models exhibit reading-order errors, hallucinations, missed content, or inaccurate table reconstruction on these examples.
  • Document understanding: MonkeyOCRv2’s richer fine-grained visual representations preserve character-level details needed to recognize dense document content.Qualitative comparisons show accurate recognition of numerical statistics, form names, and telephone numbers that general-purpose vision models handle less effectively.

6 Limitations

MonkeyOCRv2-Parsing uses a deliberately minimal frozen-encoder fine-tuning pipeline and lacks the progressive post-training used by leading specialized parsers. This limits performance on saturated benchmarks such as OmniDocBench, despite reaching open-source state-of-the-art on MDPBench.

  • Limitations: MonkeyOCRv2-Parsing freezes the encoder and uses a deliberately minimal supervised fine-tuning pipeline.The approach omits progressive post-training used by leading specialized parsers.
  • Limitations: The absence of progressive post-training leaves a performance gap to the strongest task-specific systems on saturated benchmarks such as OmniDocBench.
  • Limitations: MonkeyOCRv2-Parsing nevertheless reaches open-source state-of-the-art on the more discriminative MDPBench.

7 Conclusion

MonkeyOCRv2 is a document-oriented visual foundation model designed to preserve fine-grained textual detail and layout structure through joint text-generation and pixel-level reconstruction training. Its compact encoder family transfers across seven disparate systems, while pixel-level reconstruction improves recognition without linguistic context.

  • Model design: MonkeyOCRv2 preserves fine-grained textual detail and document layout structure unlike encoders pretrained on natural images.The model captures textual semantics while retaining character strokes, glyph shapes, and local visual details.
  • Model design: Joint autoregressive text generation and pixel-level image reconstruction align representations with textual semantics while retaining visual details.The reconstruction objective preserves character strokes, glyph shapes, and local document structure.
  • Transferability: A single document-oriented pre-training recipe transfers across seven disparate systems and improves each.The results support document intelligence encoders designed for document-specific visual statistics rather than object and scene recognition.
  • Transferability: Pixel-level reconstruction improves recognition when linguistic context is removed and narrows the performance gap.This finding indicates that reconstruction contributes visual information beyond linguistic context.

A Data Distribution of MonkeyDoc v2

MonkeyDoc v2 spans 17 languages and is dominated by English and Simplified/Traditional Chinese. Its 113M images are organized into page-level images and fine-grained cropped elements.

  • Language distribution: 17 languages are represented in MonkeyDoc v2, with English and Simplified/Traditional Chinese forming the dominant language groups.The corpus includes 19M English samples and 13M Simplified/Traditional Chinese samples.
  • Language distribution: 19M English samples and 13M Simplified/Traditional Chinese samples lead the corpus, followed by 15 other listed languages.The remaining languages are Dutch, Portuguese, Russian, French, Indonesian, Spanish, Hindi, German, Japanese, Italian, Thai, Korean, Arabic, and Vietnamese.
  • Data granularity: MonkeyDoc v2 contains 8M page-level images and 105M cropped elements across two data granularities.Page-level images support page-level data, while cropped elements support fine-grained recognition.

B Integration Protocol for Downstream Tasks

MonkeyOCRv2 backbones are integrated into five downstream document-analysis tasks by replacing task-specific visual encoders while preserving or adapting the original task heads and decoders. The protocols use task-specific feature reshaping, projection, optimization, and multiscale designs.

  • Text Recognition: 384-dimensional MonkeyOCRv2-S visual tokens replace the encoders in CRNN and PARSeq, while PARSeq retains its original decoder, autoregressive prediction, and iterative refinement.For PARSeq, tokens serve as Transformer-decoder memory for cross-attention; CRNN uses a converted two-dimensional token map.
  • Formula Recognition: 384-to-512 projection maps MonkeyOCRv2-S features to MBart hidden states after replacing UniMERNet-T’s encoder, while the original MBart decoder remains unchanged.The projection is inserted before decoder cross-attention.
  • Text Detection: 4 feature-map stages with strides 4/8/16/32 and channels [64, 128, 256, 512] integrate MonkeyOCRv2-AS into DBNet, PSENet, and DPText-DETR detectors.DBNet and PSENet use all stages through FPN necks, whereas DPText-DETR uses the last three stages in a 6-layer Transformer encoder.
  • Document Tampering Detection: 2 backbone variants replace FFDN’s ConvNeXt-V2-Base: DeepSolo-pretrained ViTAEv2-S or MonkeyOCRv2-AS, with four-scale pyramids combining RGB and frequency-enhanced features.The first two stages form the Visual Perception Head, while the deeper stages encode frequency-enhanced features.
  • Overlapping Text Segmentation: 1 × 10−4 learning rate and batch size 8 are used for Mask2Former, versus 2 × 10−4 and 4 for MOTS, with AdamW, weight decay 0.05, and poly scheduling.Both models train on 512 × 512 cropped inputs.

C Text Recognition on Common Benchmarks

MonkeyOCRv2-S improves text-recognition performance on six widely used regular and irregular scene-text benchmarks, despite their near-saturated status and expected modest gains.

  • C Text Recognition on Common Benchmarks: 2.3% average gain results from replacing CRNN’s original visual encoder with MonkeyOCRv2-S across six Common benchmarks.The benchmarks are ICDAR2013, SVT, IIIT5K, ICDAR2015, SVTP, and CUTE80.

D Vision Encoder Configuration on Document Understanding

The section compares vision encoders using the input resolutions and patch sizes supported during their original training, reporting their average visual-token counts across eight document-understanding benchmarks. Configurations therefore preserve encoder-specific preprocessing, including fixed-resolution and native-resolution settings.

  • Encoder configuration: SAM uses a fixed 1024 resolution, resizing both image dimensions to that supported setting.This fixed-resolution configuration follows SAM’s original training setup.
  • Encoder configuration: SigLIP 2 (naflex) preserves each original image’s resolution and aspect ratio rather than applying a fixed resolution.This configuration likewise follows the encoder’s original training input behavior.
  • Encoder configuration: All encoders use the input configurations from their original pretraining, including encoder-specific resolutions and patch sizes.Table 11 also summarizes each encoder’s average visual tokens across images from the eight evaluated benchmarks.

E Examples from MonkeyDoc v2

MonkeyDoc v2 contains document images spanning 17 languages, with visual examples covering European, Asian, Middle Eastern, and Southeast Asian languages.

  • Visual examples include Arabic, German, English, Spanish, French, and Hindi images.
  • Additional examples cover Indonesian, Italian, Japanese, Korean, Dutch, and Portuguese images.
  • The corpus also includes Russian, Thai, Vietnamese, Simplified Chinese, and Traditional Chinese images.
Loading 2607.11562v1…