PaperlayerView exact revision on arXiv

Source-linked AI summary

mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding

Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, Jingren Zhou

arXiv:2403.12895v1cs.CV

TL;DR

Existing MLLMs can recognize text in visual documents but lack broad structure understanding, limiting their handling of text-rich images. The paper introduces Unified Structure Learning with H-Reducer and structure-focused datasets, achieving state-of-the-art OCR-free results on 10 benchmarks, including gains of more than 10 points on 5/10 tasks among similar-sized models.

  • Problem

    Existing MLLMs have text recognition ability but lack general structure understanding for text-rich images, although structure information is critical to their semantics.

  • Method

    Unified Structure Learning trains structure-aware parsing and multi-grained text localization across five domains, using H-Reducer and the DocStruct4M dataset to encode spatial structure.

  • Results

    DocOwl 1.5 achieves state-of-the-art OCR-free performance on 10 visual document understanding tasks, improving similar-sized MLLMs by more than 10 points on 5/10 tasks.

  • Takeaways & Limitations

    Structure-aware learning improves MLLM representation of visually situated text and structure for visual document understanding.

  • Takeaways & Limitations

    DocOwl 1.5-Chat may still suffer from hallucinations in visual document understanding, which the paper leaves for future work.

Abstract

from arXiv · show

Structure information is critical for understanding the semantics of text-rich images, such as documents, tables, and charts. Existing Multimodal Large Language Models (MLLMs) for Visual Document Understanding are equipped with text recognition ability but lack general structure understanding abilities for text-rich document images. In this work, we emphasize the importance of structure information in Visual Document Understanding and propose the Unified Structure Learning to boost the performance of MLLMs. Our Unified Structure Learning comprises structure-aware parsing tasks and multi-grained text localization tasks across 5 domains: document, webpage, table, chart, and natural image. To better encode structure information, we design a simple and effective vision-to-text module H-Reducer, which can not only maintain the layout information but also reduce the length of visual features by merging horizontal adjacent patches through convolution, enabling the LLM to understand high-resolution images more efficiently. Furthermore, by constructing structure-aware text sequences and multi-grained pairs of texts and bounding boxes for publicly available text-rich images, we build a comprehensive training set DocStruct4M to support structure learning. Finally, we construct a small but high-quality reasoning tuning dataset DocReason25K to trigger the detailed explanation ability in the document domain. Our model DocOwl 1.5 achieves state-of-the-art performance on 10 visual document understanding benchmarks, improving the SOTA performance of MLLMs with a 7B LLM by more than 10 points in 5/10 benchmarks. Our codes, models, and datasets are publicly available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl1.5.

1 Introduction

MLLMs perform well on general images but struggle with text-rich images because their visual representations are not optimized for textual and structural information. The paper proposes Unified Structure Learning, H-Reducer, and supporting datasets to address this gap across five domains.

  • MLLMs face challenges understanding documents, webpages, tables, and charts because their visual encoder and V2T module are trained on general image-text pairs.These components are not specifically optimized to represent textual and structural information in text-rich images.
  • Text-rich images contain diverse structures, from plain text and grids to graphical representations, making structure information important for visual document understanding.These structures occur across documents, webpages, tables, charts, posters, invoices, infographics, and reports.
  • H-Reducer aggregates neighboring visual features through convolution to preserve relative positions while producing fewer features for high-resolution document images.It is designed as an alternative to query-based modules that affect spatial information and linear V2T modules that produce more visual features.
  • Unified Structure Learning combines structure-aware parsing and multi-grained text localization tasks across five text-rich image domains.The approach is supported by the DocStruct4M training dataset.
  • DocOwl 1.5 and DocOwl 1.5-Chat achieve state-of-the-art OCR-free performance on 10 visual document understanding tasks.They improve performance by more than 10 points on 5/10 tasks among similar-sized models.

2 Related Work

Visual Document Understanding covers rich combinations of text and visual objects and includes both recognition and higher-level semantic tasks. Prior work differs in whether it relies on OCR and in how it represents document structure.

  • Visual Document Understanding targets images with rich text information, including documents, tables, charts, natural images, and webpage screenshots.Its task formats range from information extraction to visual question answering, image captioning, and natural language inference.
  • VDU models can be categorized as OCR-dependent or OCR-free according to whether they use an off-the-shelf OCR system.OCR-dependent methods align recognized text with visual inputs, whereas OCR-free methods learn from the image directly.
  • OCR-free approaches use text-recognition tasks, but some omit structure information while others target a specific representation such as webpage HTML.Donut outputs continuous text sequences, whereas Pix2Struct generates an HTML DOM tree for webpage screenshots.
  • MLLMs connect a visual encoder and an LLM through a vision-to-text module, using components such as linear layers, Q-Former, Resampler, or Abstractor.These architectures have shown strong vision understanding and open-ended conversation abilities for natural images.

3 DocOwl 1.5

DocOwl 1.5 combines high-resolution image encoding, H-Reducer, Unified Structure Learning, and a two-stage training framework to model visually situated text and structure across diverse image domains.

  • Model Architecture: DocOwl 1.5 uses a visual encoder, H-Reducer, and LLM decoder, with global and cropped images independently processed to preserve high-resolution content.A shape-adaptive cropping module creates fixed-size sub-images while a resized global image retains overall layout information.
  • Model Architecture: H-Reducer merges horizontally adjacent visual features with convolution, reducing sequence length while preserving relative spatial information for high-resolution document understanding.A fully connected layer then projects the reduced visual features into the language embedding space.
  • Unified Structure Learning: Unified Structure Learning combines structure-aware parsing and multi-grained text localization across natural images, documents, tables, charts, and webpages.The localization tasks connect specific texts with concrete image positions, complementing parsing tasks that organize text according to overall structure.
  • Unified Structure Learning: DocStruct4M ensembles publicly available datasets and constructs structure-aware text sequences or text-position pairs with diverse task instructions.Document and webpage parsing uses line feeds and spaces to encode lines and horizontal distances, while chart parsing represents chart data as Markdown tables.
  • Training Paradigm: DocOwl 1.5 trains in two stages: visual components are tuned during Unified Structure Learning, then instruction-following modules are tuned during multi-task fine-tuning.The LLM is frozen in the first stage, while the visual encoder is frozen in the second stage.

4 DocOwl 1.5-Chat

DocReason25K is an instruction-tuning set designed to improve detailed explanations for visual document understanding, addressing benchmarks that mainly use simple answers.

  • DocReason25K provides detailed explanations for text-rich image understanding alongside answers to questions from six existing benchmarks.The source questions come from DocVQA, InfoVQA, WTQ, VisualMRC, ChartQA, and TextVQA.
  • For document, table, and webpage questions, the dataset uses structure-aware text sequences as input to generate simple answers and detailed explanations.ChartQA and TextVQA instead use images as input for explanation generation.
  • A separate judging step filters ChartQA samples whose generated answers are inconsistent with the concise answers.

5 Experiments

Experiments evaluate DocOwl 1.5 across diverse OCR-free visual document understanding tasks and ablate its visual representation, structure learning, localization, and training choices. Results support H-Reducer, Unified Structure Learning, and two-stage training, while qualitative analyses show stronger structure-sensitive answers and detailed explanations with remaining hallucinations.

  • 5.2 Main Results: DocOwl 1.5 is evaluated on 10 benchmarks spanning documents, tables, charts, natural images, and webpage screenshots against OCR-free models.The comparison includes MLLMs adapted for text recognition and smaller document-understanding models.
  • 5.3 Ablation Study: Stronger general MLLM initialization and visual-encoder tuning improve performance on text-rich images.The ablation attributes the improvement to general vision-language knowledge and enhanced visual representations.
  • 5.3 Ablation Study: H-Reducer outperforms the Abstractor with fewer crops on documents, while 1x4 merging generally suits left-to-right document and table layouts better than 2x2 merging.The 1x4 setting is selected overall; 2x2 is slightly better for chart understanding in the reported comparison.
  • 5.3 Ablation Study: Unified Structure Learning substantially improves performance across domains, with further gains from textual crop-position tokens and multi-grained text localization.Structure-aware parsing provides the major improvement, while LLM tuning contributes only slightly in the ablation.
  • 5.3 Ablation Study: Two-stage training is more beneficial and efficient than one-stage joint training when Unified Structure Learning samples become large.Joint training improves DocVQA below 1M samples, but later gains become subtle and remain below two-stage training.
  • 5.5 Qualitative Results: Qualitative results show improved document, chart, and table understanding, while DocOwl 1.5-Chat produces detailed explanations but can hallucinate.Structure-aware parsing covers documents, tables, charts, and natural images; localization covers word, phrase, line, and block granularities.

6 Conclusion

The paper concludes that Unified Structure Learning combines structure-aware parsing and multi-grained localization across five text-rich image domains, supported by H-Reducer and DocStruct4M. DocOwl 1.5 achieves state-of-the-art OCR-free performance across 10 visual document understanding benchmarks.

  • 6 Conclusion: Unified Structure Learning combines structure-aware parsing and multi-grained text localization across five domains of text-rich images.The domains are documents, webpages, tables, charts, and natural images.
  • 6 Conclusion: H-Reducer aggregates horizontally neighboring visual features with convolution to preserve structure and spatial information during alignment.The module is designed to reduce visual-feature length while maintaining relative positional relationships.
  • 6 Conclusion: DocStruct4M supports structure learning through structure-aware text sequences and multi-grained text–bounding-box pairs collected from publicly available images.The dataset is constructed for the proposed Unified Structure Learning tasks.
  • 6 Conclusion: DocOwl 1.5 achieves state-of-the-art OCR-free performance on 10 visual document understanding benchmarks.The conclusion summarizes the model-level outcome of Unified Structure Learning.
  • 6 Conclusion: Qualitative examples cover detailed explanations, document and table parsing, chart parsing, natural-image parsing, text grounding, and text recognition.The examples include multiple chart types and localization granularities from word to block.
Loading 2403.12895v1…