PaperlayerView exact revision on arXiv

Source-linked AI summary

DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding

Hao Feng, Qi Liu, Hao Liu, Jingqun Tang, Wengang Zhou, Houqiang Li, Can Huang

arXiv:2311.11810v4cs.CVcs.AI

TL;DR

Document understanding models can struggle to combine high-resolution visual perception with language and world knowledge. DocPedia processes document images in the frequency domain and uses staged training; the authors report strong benchmark results, including gains over Monkey on FUNSD and SROIE.

  • Problem

    Existing document methods can struggle with high-resolution inputs or lack world knowledge, limiting the combination of document perception and comprehension.

    Source: §1 · p. 2

  • Method

    DocPedia processes document images in the frequency domain and uses text-aware pre-training followed by context-aware fine-tuning for perception and comprehension tasks.

    Source: §1 · p. 2, §3.2 · p. 5, §3.1 · p. 5

  • Results

    DocPedia performed comparably to Monkey on DocVQA, outperformed it by 16.0% on FUNSD and 15.8% on SROIE, and scored 507 on OCRBench.

    Source: §5.2 · p. 9

  • Takeaways & Limitations

    The authors report mutual benefits from jointly learning perception and comprehension tasks, supporting DocPedia as an approach to complex high-resolution documents.

    Source: §6 · p. 12

    DocPedia shows minimal InfoVQA improvement on extremely high-aspect-ratio images and currently lacks multi-page document processing and multilingual proficiency.

    Source: §5.4 · p. 12

Abstract

from arXiv · show

This work presents DocPedia, a novel large multimodal model (LMM) for versatile OCR-free document understanding, capable of parsing images up to 2,560$\times$2,560 resolution. Unlike existing work either struggle with high-resolution documents or give up the large language model thus vision or language ability constrained, our DocPedia directly processes visual input in the frequency domain rather than the pixel space. The unique characteristic enables DocPedia to capture a greater amount of visual and textual information using a limited number of visual tokens. To consistently enhance both perception and comprehension abilities of our model, we develop a dual-stage training strategy and enrich instructions/annotations of all training tasks covering multiple document types. Extensive quantitative and qualitative experiments conducted on various publicly available benchmarks confirm the mutual benefits of jointly learning perception and comprehension tasks. The results provide further evidence of the effectiveness and superior performance of our DocPedia over other methods.

1 Introduction

Document understanding must handle intertwined visual and textual content in complex, high-resolution documents, while existing approaches can constrain either visual detail or language capability. DocPedia processes images in the frequency domain and combines high-resolution perception with an LLM, using a dual-stage training strategy.

1 Introduction

  • Document understanding combines perception and comprehension of visual and textual content in complex documents, supporting applications such as information extraction and visual question answering.The section describes sparse or dense text interwoven with graphics and tables.Source: §1 · p. 1
  • OCR-based pipelines can accumulate errors, while existing methods may constrain high-resolution perception, language capability, or fine-grained localization.Vision-constrained systems use lower input resolutions; language-constrained systems may lack LLM reasoning and world knowledge, while cropping can create semantic discontinuities.Source: §1 · pp. 1–2
  • DocPedia processes document images up to 2,560×2,560 in the frequency domain, capturing more visual and textual information with limited visual tokens.The model combines this frequency-domain input with the world knowledge and inference capabilities of LLMs.Source: §1 · pp. 2–3
  • Its training uses text-aware pre-training followed by context-aware fine-tuning, with perception and comprehension tasks learned jointly.The authors report mutual benefits from jointly learning these tasks.Source: §1 · pp. 2–3
  • DocPedia is reported to achieve superior performance on multiple public benchmarks, with experiments supporting the effectiveness of the approach.The introduction summarizes the results without naming benchmark scores.Source: §1 · p. 3

2 Related work

The related work distinguishes OCR-driven approaches from OCR-free methods. It notes OCR error accumulation and continuing challenges in handling high-resolution inputs while retaining world knowledge.

2.1 OCR-driven document understanding

  • OCR-driven methods integrate extracted text with layout and visual features, but OCR dependence can accumulate errors amid intertwined document content.The section cites LayoutLM, Wukong-Reader, StrucTexT, and DocFormer as examples.Source: §2.1 · p. 3

2.2 OCR-free document understanding

  • OCR-free models generate token sequences for varied responses and structured information, offering versatility in content creation and inquiry response.The section describes methods spanning visual instruction tuning, document parsing, and high-resolution image processing.Source: §2.2 · pp. 3–4
  • Many OCR-free methods still struggle with high-resolution inputs or lack world knowledge, motivating systems that handle diverse document types and resolutions.The section identifies lower-resolution visual encoders as a constraint for some methods.Source: §2.2 · p. 4

3 Method

DocPedia converts document images into frequency-domain features and feeds aligned visual tokens to a language model. It uses text-aware pre-training followed by context-aware fine-tuning to develop perception and comprehension.

3.1 Architecture

  • Images are resized with aspect ratio preserved, transformed into JPEG DCT coefficients, and processed into frequency features before visual tokens are aligned with language tokens for Vicuna-7B.The Swin Transformer produces features downsampled by a factor of 1/64; the visual tokens are concatenated with instruction tokens and passed to the LLM.Source: §3.1 · pp. 4–5

3.2 Text-aware pre-training

  • Text-aware pre-training freezes the LLM while training the vision encoder and projector on detection, recognition, spotting, paragraph reading, full-text reading, and image captioning.The first five tasks use document images, while image captioning targets natural scenes.Source: §3.2 · p. 5

3.3 Context-aware fine-tuning

  • Context-aware fine-tuning unfreezes the LLM and fine-tunes the whole model, splitting each batch between five OCR tasks and higher-level document and scene understanding.The method combines lower-level perception training with semantic understanding tasks.Source: §3.3 · pp. 5–6

4 Dataset construction

The training dataset combines document and natural-scene images with perception, question-answering, and instruction-tuning data. Pre-training supplies OCR-task examples, while fine-tuning adds benchmark and natural-scene data with expanded responses.

4.1 Pre-training

  • Pre-training uses 600,000 PowerPoint images from Common Crawl and 325,000 PDF images from arXiv for perceptual learning.Each image is assigned a randomly selected OCR task type, with varied instructions and standardized responses; 595,000 LLaVA caption entries add natural-scene data.Source: §4.1 · p. 6

4.2 Fine-tuning

  • Fine-tuning reuses the 325,000 PDF and 600,000 PowerPoint images, adds 370,000 entries from visual question-answering and related benchmarks, and supplements training with 158,000 natural-scene instruction-tuning entries.GPT-3.5 expands concise benchmark answers into complete sentences.Source: §4.2 · pp. 6–7
  • Figure 4 demonstrates text identification and localization in scene and document images, alongside paragraph reading.The final example omits subsequent text readouts for display convenience.Source: Figure 4 · p. 7

5 Experiment

Experiments report DocPedia’s advantages on several high-resolution document benchmarks and examine its frequency-domain inputs, training strategy, qualitative capabilities, and scope limits.

5.1 Implementation details

  • Training used one epoch per stage on eight A100 GPUs, with batch sizes of 64 and 8 for pre-training and fine-tuning, respectively.Evaluation used accuracy, counting a response as correct when it contains the ground-truth string.Source: §5.1 · p. 7

5.2 Results

  • Qualitative examples show text localization and identification in scenes and high-resolution documents, instruction-based image understanding, and detection of small cars in general scenes.Examples also show comprehension across text sizes, fonts, and styles, and responses drawing on the LLM’s world knowledge.Source: §5.2 · pp. 7–8, Figure 5 · p. 8, Figure 6 · p. 9
  • On FUNSD and SROIE, DocPedia outperformed Monkey by 16.0% and 15.8%, respectively, and achieved comparable performance on DocVQA.Evaluation covered scene-text VQA, document-oriented VQA, and key information extraction benchmarks.Source: §5.2 · pp. 8–9

5.3 Ablation studies

  • At matched visual-token counts, higher-resolution frequency inputs perform better, and the response to the same academic image becomes accurate at 2,560 resolution.At the same 1,280 resolution, image inputs perform slightly better, while frequency inputs use one quarter as many visual tokens and can accommodate inputs up to 2,560×2,560.Source: §5.3 · p. 10, Figure 8 · p. 11
  • DCT outperforms RGB flattening at 1,920 and 2,560 resolutions and is reported to yield more stable, faster training convergence.The authors attribute this to DCT’s orthogonalization of frequency components and reduction of input redundancy.Source: §5.3 · pp. 10–11
  • Removing pre-training degrades performance, while ablations support joint perception-and-understanding fine-tuning and the inclusion of position-perception tasks.In the illustrated comparison, the jointly fine-tuned model answers both spotting and understanding instructions correctly, unlike the model fine-tuned only for understanding.Source: §5.3 · p. 11, Figure 9 · p. 12

5.4 Limitation discussion

  • DocPedia shows minimal improvement on InfoVQA, currently lacks multi-page document processing, and has deficient multilingual proficiency.The authors note that some InfoVQA images have extreme aspect ratios, reaching 6,400×800.Source: §5.4 · p. 12

6 Conclusion

The conclusion presents DocPedia as an OCR-free model for high-resolution document understanding that processes visual input in the frequency domain and reports strong results on public datasets.

6 Conclusion

  • DocPedia is designed for versatile OCR-free document understanding and can handle high-resolution images.The conclusion describes it as a large multimodal model.Source: §6 · p. 12
  • Frequency-domain input captures more visual and textual information with a limited number of visual tokens.The conclusion contrasts this approach with processing visual input in pixel space.Source: §6 · p. 12
  • With a dual-stage training strategy and enriched task instructions and annotations, DocPedia shows superior performance on several public datasets.Source: §6 · p. 12

Related papers

Loading 2311.11810v4…