Source-linked AI summary
mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, Qian Qi, Ji Zhang, Fei Huang
TL;DR
Existing multimodal language models show shallow OCR-free recognition but can miss fine-grained document features without in-domain training. mPLUG-DocOwl addresses this with unified instruction tuning and a dedicated LLMDoc evaluation set, achieving stronger OCR-free document understanding than existing multimodal models while retaining broader capabilities.
Problem
Existing multimodal language models can perform shallow OCR-free text recognition but may overlook fine-grained document features such as sophisticated tables and large text blocks.
Method
mPLUG-DocOwl combines document, language-only, and general vision-and-language instruction tuning with a human-evaluated LLMDoc test set.
Results
mPLUG-DocOwl achieves OCR-free state-of-the-art performance on multiple document understanding datasets and significantly better visual-text understanding than existing multimodal models on LLMDoc.
Takeaways & Limitations
Unified instruction tuning supports stronger document instruction understanding across diverse scenarios, including complex visual-text interactions.
Takeaways & Limitations
Human evaluation shows remaining difficulties with document-related commonsense reasoning, mathematical calculations, and creative generation.
Abstract
from arXiv · showhide
Document understanding refers to automatically extract, analyze and comprehend information from various types of digital documents, such as a web page. Existing Multi-model Large Language Models (MLLMs), including mPLUG-Owl, have demonstrated promising zero-shot capabilities in shallow OCR-free text recognition, indicating their potential for OCR-free document understanding. Nevertheless, without in-domain training, these models tend to ignore fine-grained OCR features, such as sophisticated tables or large blocks of text, which are essential for OCR-free document understanding. In this paper, we propose mPLUG-DocOwl based on mPLUG-Owl for OCR-free document understanding. Specifically, we first construct a instruction tuning dataset featuring a wide range of visual-text understanding tasks. Then, we strengthen the OCR-free document understanding ability by jointly train the model on language-only, general vision-and-language, and document instruction tuning dataset with our unified instruction tuning strategy. We also build an OCR-free document instruction understanding evaluation set LLMDoc to better compare models' capabilities on instruct compliance and document understanding. Experimental results show that our model outperforms existing multi-modal models, demonstrating its strong ability of document understanding. Besides, without specific fine-tuning, mPLUG-DocOwl generalizes well on various downstream tasks. Our code, models, training data and evaluation set are available at https://github.com/X-PLUG/mPLUG-DocOwl.
1 Introduction
mPLUG-DocOwl extends mPLUG-Owl with unified instruction tuning for OCR-free document understanding. It introduces the LLMDoc evaluation set and reports stronger performance than existing multimodal models.
- mPLUG-DocOwl is a modularized MLLM built on mPLUG-Owl for OCR-free document understanding.
- mPLUG-DocOwl achieves OCR-free state-of-the-art performance across multiple commonly used document understanding datasets.
- The model balances language-only, general vision-and-language, and document understanding through unified instruction tuning.
- LLMDoc is a human-evaluated instruction understanding test set designed to assess diverse document understanding capabilities.
2 Related Work
Prior multimodal document-understanding methods either use OCR systems or learn text recognition with high-resolution encoders. Other MLLMs use tool collaboration or end-to-end multimodal architectures.
- OCR-based approaches recognize image text with off-the-shelf OCR models or APIs before aligning visual and textual inputs.
- End-to-end approaches use high-resolution image encoders to learn text recognition during pretraining.
- Systematic collaboration approaches connect LLMs with vision experts or tools that express visual information as text descriptions.
3 mPLUG-DocOwl
mPLUG-DocOwl combines a modular visual-language architecture with document instruction tuning across diverse task formats. Training freezes major foundation modules while adapting the visual abstractor and language-model LoRA.
- Architecture: mPLUG-DocOwl uses a visual foundation model, visual abstractor, and language model to convert image features and text into responses.
- Instruction Tuning Data: Document instruction tuning covers VQA, information extraction, natural language inference, and image captioning.
- Instruction Tuning Data: Information extraction is reformulated as questions asking for values associated with stationary keys, with nonexistent keys mapped to “None”.
- Instruction Tuning Data: Natural language inference is converted into Yes-or-No questions representing Entailed and Refuted labels, using TabFact.
- Instruction Tuning Data: The training data also includes language-only and general vision-and-language instruction tuning to improve language comprehension and multimodal conversation.
- Training: Training uses two stages: first adapting the visual abstractor and LoRA on document data, then freezing the abstractor while training LoRA with broader instruction data.
4 Experiment
The experiments evaluate mPLUG-DocOwl on LLMDoc and OCR-free document-understanding benchmarks using human ratings, quantitative comparisons, and qualitative cases. Results show stronger instruction understanding and broad document-processing abilities, while failure cases remain on challenging multi-step and instruction-sensitive tasks.
- LLMDoc: LLMDoc evaluates open-ended document instruction understanding with human ratings across diverse document scenarios.The evaluation set samples tables, charts, documents, natural images, and webpages, and uses an A–D response scale.
- LLMDoc: 37 responses received an “A” rating for mPLUG-DocOwl, indicating significantly better LLMDoc performance than mPLUG-Owl and Mini-GPT4.The authors also report that all compared models still receive some C or D ratings.
- Benchmark evaluation: mPLUG-DocOwl achieves comparable or better performance than existing OCR-free methods without fine-tuning each dataset.The benchmark comparisons cover DUE-Benchmark plus chart, natural-image, and webpage understanding datasets.
- Benchmark evaluation: Qualitative cases show accurate webpage retrieval, handwritten-table understanding, chart summarization, scanned-document processing, name distinction, and intent-following calculations.The examples also include an illusory chart output, showing that strong capabilities coexist with occasional errors.
- Qualitative analysis: Comparisons on LLMDoc show better table-to-JSON conversion, chart responses, and webpage summarization than Mini-GPT4, although errors remain.mPLUG-DocOwl misses a table row and misunderstands subject-object relations in one mail example, while Mini-GPT4 often fails more broadly.
- Qualitative analysis: Failure cases include ignoring the user as a speaker, failing multi-step calculations, hallucinating characters, and misreading news-writing instructions.These examples indicate that document instruction understanding remains challenging even for mPLUG-DocOwl.
5 Conclusion
mPLUG-DocOwl brings diverse OCR-free document understanding into mPLUG-Owl through document instruction fine-tuning and benefits from language-only and general vision-and-language tuning. However, it still struggles with several document-related capabilities.
- mPLUG-DocOwl infuses diverse OCR-free document understanding capabilities into mPLUG-Owl through document instruction fine-tuning.
- Language-only and general vision-and-language instruction tuning help mPLUG-DocOwl better comprehend user instructions and intentions, enabling more complex interactions.
- Human evaluation on LLMDoc reveals continued struggles with document-related commonsense reasoning, mathematical calculations, and creative generation.