Source-linked AI summary
Multimodal OCR: Parse Anything from Documents
Handong Zheng, Yumeng Li, Kaile Zhang, Liang Xin, Guangwei Zhao, Hao Liu, Jiayu Chen, Jie Lou, Qi Fu, Rui Yang, Shuo Jiang, Weijian Luo, Weijie Su, Weijun Zhang, Xingyu Zhu, Yabin Li, Yiwei ma, Yu Chen, Yuqiu Ji, Zhaohui Yu, Guang Yang, Colin Zhang, Lei Zhang, Yuliang Liu, Xiang Bai
TL;DR
Document parsing often discards the structure and semantics carried by graphics, leaving supervision from charts, diagrams, and other visual elements unused. MOCR addresses this gap by jointly parsing textual and visual elements into structured outputs, and dots.mocr achieves strong results across document parsing and image-to-SVG reconstruction. The approach also provides a route to constructing multimodal image–code–text corpora from existing documents.
Problem
Existing document parsing pipelines focus on text while treating graphics as cropped pixels, discarding structural and semantic information from documents.
Method
MOCR parses textual and visual elements jointly, using symbolic text representations and renderable SVG code for eligible graphics within a scalable, staged training system.
Results
dots.mocr performs strongly across document parsing and structured graphics reconstruction, ranking second to Gemini 3 Pro on OCR Arena Elo and surpassing it on image-to-SVG benchmarks.
Takeaways & Limitations
MOCR provides a pipeline for converting document graphics into reusable image–code–text supervision for multimodal pretraining.
Takeaways & Limitations
The current release requires separate page-level text parsing and region-level image-to-SVG decoding passes rather than one-pass full-page multimodal parsing.
Abstract
from arXiv · showhide
We present Multimodal OCR (MOCR), a document parsing paradigm that jointly parses text and graphics into unified textual representations. Unlike conventional OCR systems that focus on text recognition and leave graphical regions as cropped pixels, our method, termed dots.mocr, treats visual elements such as charts, diagrams, tables, and icons as first-class parsing targets, enabling systems to parse documents while preserving semantic relationships across elements. It offers several advantages: (1) it reconstructs both text and graphics as structured outputs, enabling more faithful document reconstruction; (2) it supports end-to-end training over heterogeneous document elements, allowing models to exploit semantic relations between textual and visual components; and (3) it converts previously discarded graphics into reusable code-level supervision, unlocking multimodal supervision embedded in existing documents. To make this paradigm practical at scale, we build a comprehensive data engine from PDFs, rendered webpages, and native SVG assets, and train a compact 3B-parameter model through staged pretraining and supervised fine-tuning. We evaluate dots.mocr from two perspectives: document parsing and structured graphics parsing. On document parsing benchmarks, it ranks second only to Gemini 3 Pro on our OCR Arena Elo leaderboard, surpasses existing open-source document parsing systems, and sets a new state of the art of 83.9 on olmOCR Bench. On structured graphics parsing, our model achieves higher reconstruction quality than Gemini 3 Pro across image-to-SVG benchmarks, demonstrating strong performance on charts, UI layouts, scientific figures, and chemical diagrams. These results show a scalable path toward building large-scale image-to-code corpora for multimodal pretraining. Code and models are publicly available at https://github.com/rednote-hilab/dots.mocr.
1 Introduction
MOCR broadens document parsing beyond text by treating graphics as structured parsing targets, addressing information loss from raster-only handling. The scalable dots.mocr system combines multimodal data and staged training to achieve strong performance across document and graphics parsing.
- Motivation: Existing OCR pipelines often crop non-textual elements as pixels, discarding structural and semantic information from document graphics.This makes document parsing inherently lossy and limits recoverable supervision.
- MOCR paradigm: MOCR parses text, layout, tables, charts, diagrams, icons, and UI components as first-class elements, producing reusable structured outputs.Graphics are represented as renderable SVG code alongside textual content.
- Challenges: MOCR faces scarce aligned graphics supervision, non-unique renderable programs, and the combined difficulty of visual grounding and long-sequence generation.These challenges make scalable graphics parsing substantially harder than text-only OCR.
- System: dots.mocr uses data from PDFs, rendered webpages, and native SVG graphics with staged OCR and graphics-centric training, normalization, and quality control.The design aligns predicted code with faithful rendering within one unified architecture.
- Implications: MOCR recovers graphics as reusable, renderable code rather than raster crops, creating structured supervision from existing documents.This is presented as a new source of supervision for downstream reasoning and multimodal pretraining.
- Results: dots.mocr ranks second to Gemini 3 Pro on OCR Arena Elo, sets a state of the art on olmOCR-Bench, and surpasses Gemini 3 Pro on image-to-SVG benchmarks.The compact 3B-parameter model performs strongly across document parsing and structured graphics reconstruction.
2 Related Work
Related work spans text-centric document parsing and structured graphics parsing, while MOCR connects these directions by converting diverse document elements into reusable, renderable representations.
- Text Parsing: Text parsing methods extract content from formats including PDFs, webpages, slides, spreadsheets, scans, and scene-text images, often using vision-language models.Existing approaches include multi-stage pipelines combining layout analysis, detection, recognition, and reading-order prediction.
- Structured Graphics Parsing: Structured graphics parsing recovers layout, geometry, styling, and spatial relations as executable representations such as HTML, LaTeX, SVG, or Python.Website and UI parsing translate screenshots into DOM-like structures or front-end code.
- MOCR: MOCR converts text, charts, diagrams, UI elements, icons, and domain drawings into reusable, renderable representations instead of raster crops.It bridges text-focused parsers and task-specific graphic systems while targeting executable supervision for multimodal pretraining and retrieval.
3 Multimodal OCR
MOCR unifies page-level parsing of text, layout, and visual symbols, representing eligible graphics as reusable structured code rather than raster crops. The system combines a high-resolution vision encoder, multimodal connector, language decoder, and staged data-driven training, while its current release uses separate passes for page parsing and image-to-SVG decoding.
- Multimodal OCR: MOCR unifies document, webpage, UI, scene-text, and structured-graphics parsing within one model.
- Task Definition: Each parsed element contains a bounding box, semantic category, and type-specific payload generated in human-centric reading order.Text regions use symbolic transcriptions, while eligible visual symbols use renderable SVG representations.
- Task Definition: MOCR converts charts, diagrams, icons, and UI components into reusable, renderable representations, while retaining complex imagery as raster content.This supports render-and-reuse workflows and provides granular structural supervision for multimodal pretraining.
- Task Definition: The current release does not produce full-page parsing and visual-symbol parsing in one pass; it completes the multimodal parse through separate task-conditioned passes.
- Model Architecture: The architecture combines a high-resolution vision encoder, lightweight multimodal connector, and autoregressive language-model decoder.The encoder accepts inputs up to approximately 11M pixels, while the decoder generates heterogeneous, long structured outputs such as SVG programs.
- Training and Data Engine: The data engine draws on PDFs, rendered webpages, native SVG assets, and general-purpose data, supporting staged pretraining and supervised instruction tuning.Quality control and normalization align structured code with faithful rendering and improve output consistency.
4 Experiments
The experiments evaluate document parsing, structured graphics reconstruction, and general vision-language capability using shared comparison protocols. dots.mocr performs strongly across these settings, while retaining headroom in some document categories.
- Document Parsing: dots.mocr achieves the strongest Elo performance among the listed open-source models across all three document parsing benchmarks, while Gemini 3 Pro ranks first.All models are evaluated under the same Elo-style paired-comparison protocol with Gemini 3 Flash as judge.
- Evaluation: Elo ratings provide complementary signals to strict-match metrics because pairwise judging reduces sensitivity to formatting differences and captures error severity on common pages.The ratings are not expected to numerically align with strict-match metrics or existing leaderboard scores.
- Document Parsing: dots.mocr achieves the best overall score on olmOCR-Bench and the highest scores in ArXiv, Old scans math, Tables, and Multi column categories.Other systems lead in Old scans, Headers & footers, Long tiny text, and Base, indicating remaining headroom.
- Structured Graphics Parsing: The graphics evaluation spans vector graphics, charts, webpage and UI layouts, exam diagrams, scientific figures, and chemistry structure diagrams.Each prediction is rendered and compared with the original image using the ISVGEN score.
- Structured Graphics Parsing: dots.mocr-svg surpasses OCRVerse by +0.139 on UniSVG overall, scoring 0.902 versus 0.763, and outperforms Gemini 3 Pro on all reported downstream benchmarks.Improvements are especially clear on structure-sensitive ChartMimic and ChemDraw tasks, while remaining strong on layouts and scientific figures.
- General Capability: dots.mocr remains highly competitive on broader vision-language benchmarks and shows clear advantages on CharXiv in descriptive and reasoning settings.The evaluation covers document understanding and multimodal reasoning beyond the primary parsing tasks.
5 Qualitative examples
Qualitative examples show that dots.mocr and dots.mocr-svg produce structured, reusable representations across heterogeneous documents and visual domains. Their outputs capture both layout structure and editable vector-level content.
- Qualitative examples: dots.mocr generates layout partitions and executable SVG code instead of raster crops, yielding reusable and machine-readable outputs.Examples cover heterogeneous documents, multilingual pages, and complex multi-column layouts.
- Qualitative examples: dots.mocr-svg reconstructs icons, statistical charts, and cross-disciplinary illustrations as editable, scalable SVG representations.The examples include bar, line, scatter, and composite charts, with geometric structure and semantic grouping preserved.
- Qualitative examples: The model retains strong document parsing and SVG reconstruction while producing coherent, context-aware responses across documents, charts, UI screenshots, and complex illustrations.
6 Discussion
The discussion frames MOCR as both a data-construction pipeline and an evaluation approach for increasingly structured multimodal outputs. It identifies image–code corpora and render-based quality control as important extensions.
- Discussion: MOCR can convert document graphics into image–code pairs and image, code, and text triples for scalable multimodal pretraining corpora.Charts and diagrams become controllable and perturbable training data derived from existing documents.
- Discussion: Normalization and render-based verification address non-unique program targets, while tighter verification loops, reward-model filtering, and self-improving curation offer paths for further improvement.The discussion also presents OCR Arena as an alternative to brittle rule-based metrics for structurally diverse outputs.
7 Conclusion
MOCR broadens document parsing from text extraction to structured understanding of all information-bearing elements. By recovering graphics as reusable, renderable code, it converts discarded visual content into structured supervision.
- Conclusion: MOCR treats text, graphics, charts, diagrams, and icons as first-class parsing targets rather than opaque raster regions.The paradigm recovers visual symbols as reusable, renderable structured code.
- Conclusion: Recovering graphics as executable and re-renderable SVG programs transforms static document pixels into structured supervision for reasoning and learning.The conclusion states that this increases the usable supervision extractable from large document corpora.
A.1 Other Benchmarks
Table 6 compares general-purpose and specialized VLMs on OmniDocBench v1.5 and pdf-parse-bench. dots.mocr performs strongest on text transcription and reading order, while formula and table metrics require cautious interpretation.
- Evaluation caveats: OmniDocBench v1.5 Formula and Table metrics are omitted because detection rules and ground-truth matching protocols can introduce substantial variance.The evaluation aggregates metrics from OmniDocBench v1.5 and corresponding model publications, while pdf-parse-bench results are reproduced under a unified setup.
- Results: dots.mocr achieves the strongest performance on OmniDocBench v1.5 text transcription and reading-order metrics.The reported metrics are TextEdit and ReadOrderEdit.
- Results: dots.mocr improves over dots.ocr on formula-intensive pages.The authors report this as an observed improvement in mathematical-content recognition.
A.2 Qualitative examples
This section presents qualitative examples of outputs produced by dots.mocr and dots.mocr-svg in Figures 4–9.
- Qualitative examples: Qualitative examples of dots.mocr outputs are presented in Figures 4–9.The examples are introduced as visual demonstrations of the models' outputs.
- Qualitative examples: The qualitative examples cover outputs from both dots.mocr and dots.mocr-svg.The section presents examples from the document-parsing and SVG-parsing systems.
- Qualitative examples: Figures 4–9 provide visual output examples rather than a separate quantitative evaluation.The passage characterizes the section as presenting qualitative examples.
A.3 OCR Arena
The OCR Arena evaluation uses an impartial VLM judge to compare Markdown OCR outputs against the same source image, while Figures 4–9 illustrate parsing across documents and structured graphics. The examples span layouts, webpages, icons, charts, scientific illustrations, and broader vision–language tasks.
- OCR Arena: An impartial high-capacity VLM compares two Markdown OCR outputs conditioned on the same source document image.The judging prompt appears in Figure 10, with example judging results in Figure 11.
- Document parsing: dots.mocr identifies titles, paragraphs, columns, tables, formulas, scanned text, and handwriting across heterogeneous documents.Figure 4 presents qualitative layout analysis results across diverse formats and languages.
- Document parsing: dots.mocr preserves global reading order in long webpages and organizes scene text in complex real-world images.Figure 5 highlights generalization beyond conventional OCR benchmarks.
- Structured graphics parsing: dots.mocr-svg converts raster icons into concise executable SVG code with geometric primitives, grouping, and spatial relationships.Figure 6 shows rendered SVG parsing outputs and reconstructions for icon images.
- Structured graphics parsing: dots.mocr-svg recovers chart structure, axes, legends, data encodings, and semantic grouping for diverse statistical visualizations.Figure 7 includes bar, line, scatter, and composite charts represented as executable SVG.
- Broader vision–language tasks: dots.mocr produces coherent context-aware answers across documents, charts, UI screenshots, and complex illustrations.Figure 9 presents general-purpose visual question answering results alongside the model's parsing capabilities.