PaperlayerView exact revision on arXiv

Source-linked AI summary

mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding

Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, Jingren Zhou

arXiv:2409.03420v2cs.CV

TL;DR

High-resolution document understanding can require thousands of visual tokens, slowing inference and burdening multi-page comprehension. DocOwl2 compresses each high-resolution page with global visual guidance and reports state-of-the-art multi-page results alongside reduced first-token latency.

  • Problem

    High-resolution OCR-free document models can generate thousands of visual tokens per page, increasing inference time and GPU memory use and limiting multi-page comprehension.

    Source: §1 · p. 2

  • Method

    DocOwl2 uses global low-resolution visual features to guide cross-attention compression of high-resolution document images into 324 tokens, with a three-stage training framework.

    Source: §1 · p. 3, §5 · p. 15

  • Results

    DocOwl2 reports OCR-free state-of-the-art results on two multi-page document benchmarks and one text-rich video benchmark, with <50% First Token Latency on multi-page benchmarks.

    Source: §1 · p. 3, §5 · p. 15

  • Takeaways & Limitations

    For multiple images, DocOwl2 can answer questions with detailed explanations and evidence, and can identify unanswerable questions with reasons.

    Source: §4.4 · p. 15

    The compression rationale assumes that post-vision-to-text visual features can be further compressed while retaining most visually situated textual information.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Multimodel Large Language Models(MLLMs) have achieved promising OCR-free Document Understanding performance by increasing the supported resolution of document images. However, this comes at the cost of generating thousands of visual tokens for a single document image, leading to excessive GPU memory and slower inference times, particularly in multi-page document comprehension. In this work, to address these challenges, we propose a High-resolution DocCompressor module to compress each high-resolution document image into 324 tokens, guided by low-resolution global visual features. With this compression module, to strengthen multi-page document comprehension ability and balance both token efficiency and question-answering performance, we develop the DocOwl2 under a three-stage training framework: Single-image Pretraining, Multi-image Continue-pretraining, and Multi-task Finetuning. DocOwl2 sets a new state-of-the-art across multi-page document understanding benchmarks and reduces first token latency by more than 50%, demonstrating advanced capabilities in multi-page questioning answering, explanation with evidence pages, and cross-page structure understanding. Additionally, compared to single-image MLLMs trained on similar data, our DocOwl2 achieves comparable single-page understanding performance with less than 20% of the visual tokens. Our codes, models, and data are publicly available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2.

1 INTRODUCTION

High-resolution document models can consume many visual tokens, burdening inference and limiting multi-page contexts. DocOwl2 uses layout-aware compression and reports strong multi-page and single-page results with reduced token use.

1 INTRODUCTION

  • High-resolution document models can generate thousands of visual tokens per image, increasing inference time and GPU memory use and hindering multi-page understanding.InternVL 2 averages 3k visual tokens on DocVQA, according to the introduction.Source: §1 · p. 2
  • Existing compression methods either retain long sequences after compressing crops or overlook overall layout when using learnable queries or selected tokens.Layout-aware guidance matters because texts within a layout region are semantically coherent.Source: §1 · p. 2
  • DocOwl2 places a cross-attention High-resolution DocCompressor after its vision-to-text module and trains through single-image pretraining, multi-image continue-pretraining, and multi-task finetuning.Global low-resolution visual features guide compression; relative-position high-resolution features are collected as compressing objects.Source: §1 · pp. 2–3
  • High-resolution DocCompressor outperforms existing compressing methods on OCR-free single-image document understanding with fewer visual tokens.Source: §1 · p. 3
  • DocOwl2 achieves state-of-the-art performance on multi-page document benchmarks with <50% First Token Latency.The introduction reports both results for the multi-page setting.Source: §1 · p. 3
  • On 10 single-image document benchmarks, DocOwl2 achieves comparable performance with <20% visual tokens versus state-of-the-art MLLMs of similar size and training data.Source: §1 · p. 3

2 RELATED WORK

OCR-free document models encode text-rich pages with varied image-processing strategies, but high-resolution approaches often use many visual tokens. Visual-feature compression methods seek to reduce tokens while retaining useful document information.

2.1 OCR-FREE VISUAL DOCUMENT UNDERSTANDING

  • OCR-free visual document understanding spans scans, infographics, charts, table images, webpages, and natural images with scene text.mPLUG-DocOwl and UReader unified tasks across five types of document images in a sequence-to-sequence format.Source: §2.1 · p. 3
  • High-resolution OCR-free methods use cropping, sliding windows, higher-resolution encoders, or frequency-domain processing, but commonly produce over 1k visual tokens per A4 page.The section describes approaches including UReader, Monkey, mPLUG-DocOwl1.5, DocPedia, CogAgent, InternLM-XComposer, and InternVL.Source: §2.1 · p. 4

2.2 VISUAL FEATURE COMPRESSING

  • General-image query-based compressors can miss rich document text, while crop-wise compression still exceeds 1k tokens; DocOwl2 uses global-image queries and sub-image features as keys and values.The passage says this produces a fixed number of visual tokens for any resolution and covers all areas during compression.Source: §2.2 · p. 4

3 MPLUG-DOCOWL2

DocOwl2 processes each page through shape-adaptive cropping, high-resolution visual encoding, and compression, then combines compressed page features with instructions for an LLM.

3 MPLUG-DOCOWL2

  • DocOwl2 processes each image independently through shape-adaptive cropping, high-resolution visual encoding, and a high-resolution compressor.Compressed features from multiple pages are then combined with text instructions for multimodal understanding.Source: §3 · p. 4, Figure 3 · p. 5

3.1 HIGH RESOLUTION VISION ENCODING

  • Shape-adaptive cropping chooses R × C from image resolution and resizes the global image and crops to H × W; ViT and H-Reducer encode and align their features to the LLM dimension.Source: §3.1 · p. 5

3.2 HIGH RESOLUTION FULL-COMPRESSING

  • DocOwl1.5 uses 2,560 tokens at 1,344 × 1,344; global-image queries attend to aligned crop features, compressing output to the global image's token count.Source: §3.2 · p. 6

3.3 MULTI-IMAGE MODELING WITH LLM

  • Compressed page features are concatenated with text instructions for the LLM, and an <img x> ordinal marker identifies each image's order.Source: §3.3 · pp. 6–7

3.4 MODEL TRAINING

  • Training proceeds through single-image pretraining, multi-image continued pretraining, and multitask finetuning on single- and multi-image datasets.Multi-image training includes page parsing and lookup; finetuning includes evidence-page explanations and hierarchical scientific-paper text.Source: §3.4 · p. 7

4.1 IMPLEMENTATION DETAILS

DocOwl2 uses three training stages that progressively move from single-image pretraining to multi-image continuation and multi-task finetuning.

4.1 IMPLEMENTATION DETAILS

  • Single-image Pretraining trains the vision encoder, H-Reducer, and High-resolution DocCompressor while freezing the main LLM parameters and tuning MAM.This stage runs 12k steps with batch size 1,024 and learning rate 1e-4; the vision encoder starts from ViT/L-14 initialization.Source: §4.1 · p. 8
  • Multi-image Continue-pretraining freezes the vision encoder and tunes H-Reducer, High-resolution DocCompressor, and MAM for 2.4k steps.The batch size is 1,024 and the learning rate is 2e-5.Source: §4.1 · p. 8
  • Multi-task Finetuning optimizes all parameters except the vision encoder for 9k steps, using batch size 256 and learning rate 2e-5.The maximum crop count is 12, each sub-image or global image is 504x504, and the compressor has two cross-attention layers.Source: §4.1 · p. 8

4.2 MAIN RESULTS

DocOwl2 is evaluated on single-image, multi-page, and text-rich video understanding, with comparisons covering question-answering performance and inference latency. The reported results indicate strong performance with fewer visual tokens, including across multi-page benchmarks.

4.2 MAIN RESULTS

  • The evaluation compares question-answering performance and First Token Latency across 10 single-image, 2 multi-page, and 1 text-rich video benchmark.Table 3 and Table 4 report First Token Latency in seconds; Table 4 also defines TokenV as average visual tokens per page or frame.Source: §4.2 · pp. 8–9, Table 3 · p. 9, Table 4 · p. 10

4.2.1 SINGLE-IMAGE DOCUMENT UNDERSTANDING

  • Across 10 single-image benchmarks, DocOwl2 achieves better or comparable performance to models using fewer than 1k visual tokens and exceeds 80% performance on 7/10 benchmarks with fewer than 20% of the tokens used by models above 1k.Figure 4 compares OCR-free performance and average visual-token counts across the 10 benchmarks.Source: §4.2.1 · p. 9, Figure 4 · p. 9
  • DocOwl2 achieves 98% of DocOwl 1.5 performance while reducing First Token Latency by 50% with 20% of its visual tokens.The comparison uses similar OCR-learning training data and covers documents, charts, and natural images.Source: §4.2.1 · pp. 9–10

4.2.2 MULTI-PAGE/VIDEO DOCUMENT UNDERSTANDING

  • With fewer visual tokens per image, DocOwl2 achieves better question-answering performance and much less First Token Latency than multi-page/video baselines.The comparison covers multi-page document and text-rich video benchmarks.Source: §4.2.2 · p. 10, Table 4 · p. 10

4.3 ABLATION STUDY

The ablations examine the compressor architecture and three-stage training strategy, reporting effects on single-image and multi-page understanding. They also describe how compressor guidance, placement, attention, and input resolution relate to performance.

4.3.1 COMPRESSOR ARCHITECTURE

  • High-resolution DocCompressor outperforms CAbstractor, while Resampler performs worse than CAbstractor, supporting global visual features as layout-aware compression guidance.The comparison uses an identical single-image pretraining and finetuning pipeline with consistent data and settings.Source: §4.3.1 · p. 10
  • Placing the compressor before H-Reducer, using complete attention across sub-image tokens, or mean-pooling feature groups each underperforms the proposed configuration.The authors attribute the placement result to textual-semantic retention after vision-to-text conversion and the attention result to positional correspondence as a reliable prior.Source: §4.3.1 · p. 11
  • Reducing cross-attention layers slightly improves DocVQA but worsens WikiTablesQA, while four layers add no significant improvement; increasing crop count or base resolution benefits performance.The authors report larger improvement from increased base resolution because it provides more visual tokens after compression.Source: §4.3.1 · p. 11

4.3.2 TRAINING STRATEGY

  • Ensembling single-image and multi-image instruction-tuning sets for Multi-task Finetuning yields the best single-page and multi-page benchmark performance.Multi-image Continue-pretraining particularly improves performance on documents exceeding 10 pages; single-image-only training struggles beyond 10 pages.Source: §4.3.2 · p. 11, Table 6 · p. 11
  • Figures 5–7 show qualitative results: Figure 5 presents DocOwl2 multi-page text lookup and parsing after Multi-image Continue Pretraining, Figure 6 detailed-explanation question answering, and Figure 7 text-rich video understanding.Source: Figure 5 · p. 12, Figure 6 · p. 13, Figure 7 · p. 14

4.4 QUALITATIVE RESULTS

The qualitative results illustrate text parsing and lookup across document pages, evidence-based answering, and fine-grained text understanding in videos.

4.4 QUALITATIVE RESULTS

  • With 324 tokens per high-resolution image, DocOwl2 locates pages containing specified text and parses detailed text from two specified images.The authors state that 324 tokens are enough to encode detailed text information in common A4-sized pages.Source: §4.4 · p. 15
  • After Multi-task Finetuning, DocOwl2 gives an initial answer followed by a detailed explanation with evidence, and can identify unanswerable questions with reasons.The examples include PDF-rendered pages and document scans.Source: §4.4 · p. 15
  • For text-rich videos, DocOwl2 distinguishes fine-grained textual differences among similar frames, locates relevant frames, and answers accurately.Source: §4.4 · p. 15

5 CONCLUSION

mPLUG-DocOwl2 combines high-resolution document compression with OCR-free understanding across single-image, multi-page, and text-rich video tasks. It achieves state-of-the-art results on multi-page document benchmarks while using fewer visual tokens for single-image understanding.

5 CONCLUSION

  • DocOwl2 achieves OCR-free state-of-the-art performance on two multi-page document understanding benchmarks and one text-rich video understanding benchmark.Source: §5 · p. 15
  • High-resolution DocCompressor compresses each high-resolution document image into 324 tokens using cross-attention guided by global visual features and cropped-image features as keys and values.Source: §5 · p. 15
  • On single-image document benchmarks, DocOwl2 outperforms existing compression methods and performs comparably to state-of-the-art MLLMs trained on similar data with fewer visual tokens.Source: §5 · p. 15
  • Experiments indicate that thousands of visual tokens for one common A4-sized document page may be redundant and waste computational resources.Source: §5 · p. 15
Loading 2409.03420v2…