Source-linked AI summary
MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios
Zhang Li, Zhibo Lin, Qiang Liu, Ziyang Zhang, Shuo Zhang, Zidun Guo, Jiajun Song, Jiarui Zhang, Xiang Bai, Yuliang Liu
TL;DR
Document parsing lacks systematic evaluation across diverse languages, scripts, and real-world photographed documents. MDPBench introduces a 3,400-image benchmark spanning 17 languages with rigorously verified annotations and evaluates diverse parsing systems. Results show that open-source models lag behind proprietary models, while performance drops substantially on photographed documents and non-Latin scripts.
Problem
Existing benchmarks largely emphasize clean digital documents and limited languages, leaving multilingual and photographed document parsing insufficiently evaluated.
Method
MDPBench combines 3,400 document images across 17 languages with expert model labeling, manual correction, human verification, and public-private evaluation splits.
Results
Open-source models lag behind proprietary models, while performance drops by an average of 17.8% on photographed documents and 14.0% on non-Latin scripts.
Takeaways & Limitations
MDPBench provides a standardized basis for evaluating multilingual text understanding and OCR capabilities and exposes limitations in current document-parsing approaches.
Abstract
from arXiv · showhide
We introduce Multilingual Document Parsing Benchmark, the first benchmark for multilingual digital and photographed document parsing. Document parsing has made remarkable strides, yet almost exclusively on clean, digital, well-formatted pages in a handful of dominant languages. No systematic benchmark exists to evaluate how models perform on digital and photographed documents across diverse scripts and low-resource languages. MDPBench comprises 3,400 document images spanning 17 languages, diverse scripts, and varied photographic conditions, with high-quality annotations produced through a rigorous pipeline of expert model labeling, manual correction, and human verification. To ensure fair comparison and prevent data leakage, we maintain separate public and private evaluation splits. Our comprehensive evaluation of both open-source and closed-source models uncovers a striking finding: while closed-source models (notably Gemini3-Pro) prove relatively robust, open-source alternatives suffer dramatic performance collapse, particularly on non-Latin scripts and real-world photographed documents, with an average drop of 17.8% on photographed documents and 14.0% on non-Latin scripts. These results reveal significant performance imbalances across languages and conditions, and point to concrete directions for building more inclusive, deployment-ready parsing systems. Source available at https://github.com/Yuliang-Liu/MultimodalOCR.
1 Introduction
Existing document-parsing benchmarks emphasize clean digital documents and limited languages, leaving multilingual and photographed scenarios under-evaluated. MDPBench addresses this gap with a 3,400-image, 17-language benchmark and shows substantial weaknesses in current models, especially for photographed documents and non-Latin scripts.
- Existing benchmarks predominantly cover digital-born and scanned documents in limited languages, while performance declines on multilingual and photographed inputs.
- Multilingual and photographed parsing matters for accessing global knowledge and large-scale real-world documents such as archives, receipts, books, and handwritten notes.
- MDPBench contains 3,400 document images across 17 languages and diverse document types, targeting multilingual digital and photographed parsing.
- The benchmark combines expert model labeling, manual correction, and human verification to produce high-quality annotations.
- 17.8% average performance degradation occurs on photographed documents, while non-Latin-script performance decreases by 14.0%.
- Gemini-3-Pro outperforms the strongest open-source model, dots.mocr, by 7.9% in photographed scenarios.
2 Related Work
Document-parsing systems range from fixed traditional pipelines to general and specialized vision-language models. Recent benchmarks increasingly evaluate multiple tasks, but the supplied related-work passages chiefly classify methods and note the move toward unified evaluation.
- 2.1 Document Parsing Methods: Traditional document-parsing pipelines detect layout, recognize elements, merge results, and reconstruct reading order using multiple task-specific models.
- 2.1 Document Parsing Methods: PaddleOCR-VL uses a similar three-stage framework, while MinerU2.5 and MonkeyOCR v1.5 simplify parsing into two stages by merging structure detection and relation prediction.
- 2.2 Document Parsing Benchmarks: Early benchmarks targeted isolated tasks including layout analysis, formula recognition, and table recognition.
- 2.2 Document Parsing Benchmarks: Recent benchmarks shift toward unified, multi-task evaluation frameworks, including settings focused on structured academic-paper documents.
3 MDPBench
MDPBench is designed to evaluate multilingual document parsing under diverse document types and realistic photographed conditions. Its construction combines broad data collection, staged annotation, and page-level multilingual evaluation.
- Dataset construction: MDPBench targets diversity, realism, and reliability for multilingual photographed document parsing.The dataset construction covers document types, layout complexity, visual elements, and real-world capture conditions.
- Dataset construction: The benchmark sources documents across 17 representative languages and varied formats, including papers, reports, educational materials, handwritten notes, archives, and newspapers.
- Dataset construction: Photographed documents are created by capturing printed or screen-displayed documents indoors and outdoors with deformation and other visual degradations.
- Annotation pipeline: Annotations combine expert model labeling, manual correction, and independent human verification until acceptance criteria are satisfied.The verification process returns failed annotations for correction and repeats iteratively.
- Evaluation metrics: MDPBench uses page-level aggregation and evaluates text, reading order, formulas, and tables with task-specific metrics.Text and reading order use Normalized Edit Distance, formula recognition uses CDM, and table recognition uses Tree-Edit-Distance-based evaluation.
4 Experiments
Experiments show a clear gap between proprietary and open-source parsing models, alongside substantial weaknesses on photographed documents, non-Latin scripts, and language-specific structures. The analyses also identify reading-order, hallucination, recognition, and layout-related failure patterns.
- End-to-end evaluation: The strongest open-source model, dots.mocr, reaches 80.5% overall accuracy, trailing the top proprietary model.
- End-to-end evaluation: 17.8% average performance drops occur on photographed documents, while non-Latin-script languages show a 14.0% average decrease relative to Latin-script languages.
- Language-specific errors: Models produce language-specific errors including missing diacritics in Hindi, Cyrillic–Latin visual confusion in Russian, and hallucinated spaces in Thai.These errors can disrupt lexical integrity, as illustrated by incorrect segmentation of continuous Thai text.
- Reading order: Models often process two-column Arabic documents in left-to-right order despite Arabic's right-to-left reading direction.
- Single-task evaluation: PaddleOCR-VL-1.5 improves photographed-document performance by 11.6%, while MinerU-2.5-VLM retains above-85% PageIoU on Arabic, Hindi, and Russian despite low overall results.
5 Conclusion
MDPBench is a multilingual photographed-document benchmark designed to expose weaknesses in existing parsing models and support more robust evaluation and deployment.
- MDPBench contains 3,400 human-annotated images across 17 languages and diverse real-world capture conditions.
- Existing document parsing models show significant degradation on non-Latin scripts and photographed documents.
- The benchmark evaluates specialized parsing systems as well as the multilingual text understanding and OCR capabilities of general-purpose large multimodal models.
- MDPBench provides insights for improving models and developing more robust, generalizable, and practically deployable document parsing systems.