PaperlayerView exact revision on arXiv

Source-linked AI summary

LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding

Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, Cong Yao

arXiv:2404.05225v1cs.CVcs.CL

TL;DR

LLM- and MLLM-based document understanding has not fully exploited document layout, despite layout being important for precise understanding. LayoutLLM addresses this gap with layout-aware pre-training and supervised fine-tuning, including LayoutCoT, and outperforms existing LLM/MLLM methods on standard benchmarks while retaining interpretable intermediate steps. Its current limitations include absent false-positive refusal and difficulty understanding some region-level relationships.

  • Problem

    Existing LLM/MLLM document-understanding methods do not fully exploit document layout, while layout is vital for precise understanding.

  • Method

    LayoutLLM uses a document pre-trained encoder, layout-aware pre-training at document, region, and segment levels, and LayoutCoT-based layout-aware supervised fine-tuning.

  • Results

    LayoutLLM consistently and significantly improves over compared LLMs and MLLMs across evaluation benchmarks, including about 10% higher DocVQA performance than mPLUG-DocOWL and Qwen-VL.

  • Takeaways & Limitations

    LayoutCoT focuses answering on relevant document regions and provides interpretable intermediate steps that support manual inspection, correction, and interactive correction.

  • Takeaways & Limitations

    LayoutLLM currently cannot refuse false-positive outputs or generate hints, and it struggles to precisely understand region-level relationships.

Abstract

from arXiv · show

Recently, leveraging large language models (LLMs) or multimodal large language models (MLLMs) for document understanding has been proven very promising. However, previous works that employ LLMs/MLLMs for document understanding have not fully explored and utilized the document layout information, which is vital for precise document understanding. In this paper, we propose LayoutLLM, an LLM/MLLM based method for document understanding. The core of LayoutLLM is a layout instruction tuning strategy, which is specially designed to enhance the comprehension and utilization of document layouts. The proposed layout instruction tuning strategy consists of two components: Layout-aware Pre-training and Layout-aware Supervised Fine-tuning. To capture the characteristics of document layout in Layout-aware Pre-training, three groups of pre-training tasks, corresponding to document-level, region-level and segment-level information, are introduced. Furthermore, a novel module called layout chain-of-thought (LayoutCoT) is devised to enable LayoutLLM to focus on regions relevant to the question and generate accurate answers. LayoutCoT is effective for boosting the performance of document understanding. Meanwhile, it brings a certain degree of interpretability, which could facilitate manual inspection and correction. Experiments on standard benchmarks show that the proposed LayoutLLM significantly outperforms existing methods that adopt open-source 7B LLMs/MLLMs for document understanding. The training data of the LayoutLLM is publicly available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/LayoutLLM

1. Introduction

LayoutLLM addresses the difficulty of using document layout in zero-shot LLM/MLLM document understanding. It combines layout-aware pre-training with layout-aware supervised fine-tuning, including LayoutCoT, to model document information from global structure to relevant local regions.

  • Motivation: Zero-shot adaptation of document pre-trained models is difficult because they require fine-tuning on downstream task data.Here, zero-shot means not using downstream-task training sets.
  • Motivation: Plain-text representations discard layout, while coordinate-enriched layout text does not guarantee effective layout comprehension by LLMs.Layout text can introduce coordinates without producing stable performance improvements.
  • Motivation: Existing document-based MLLMs also underuse layout because common image-captioning and flattened-text pre-training provide only brief document representations.These methods typically combine a visual model with an LLM and use document pre-training followed by supervised fine-tuning.
  • Approach: LayoutLLM integrates a document pre-trained model as encoder and trains it with layout-aware pre-training plus layout-aware supervised fine-tuning.Layout-aware pre-training targets document-level, region-level, and segment-level information.
  • Approach: LayoutCoT uses Question Analysis, Relevant Area Concentration, and Answer Formation to focus on relevant regions and leverage their characteristics when answering questions.The intermediate layout-aware steps also provide a degree of interpretability and support manual intervention or correction.
  • Results: Experiments on five widely used document-understanding benchmarks demonstrate the effectiveness of LayoutLLM in zero-shot settings.The reported contribution is significant improvement over existing LLM/MLLM-based methods.

2. Related Works

Related work establishes document layout as an effective modeling signal and develops pre-training tasks for learning spatial, textual, visual, and geometric structure.

  • Document pre-trained models: Document pre-trained models jointly model text, layout, and images through diverse architectures and attention mechanisms.These studies report significant advances in document understanding by explicitly modeling layout information.
  • Layout-related pre-training tasks: Prior layout-related pre-training tasks include masked vision-language modeling, position masking, geometric pre-training, and layout-aware generation.These tasks respectively recover masked text or positions, learn geometric relations, or generate structured text with layout information.

3. LayoutLLM

LayoutLLM combines a document-pretrained encoder with layout instruction tuning to improve document-layout comprehension. Its training uses multi-level layout-aware pretraining and LayoutCoT-based supervised fine-tuning to guide question answering through relevant regions.

  • 3.2. Layout Instruction Tuning: The model is trained in two stages: layout-aware pretraining and layout-aware supervised fine-tuning.
  • 3.1. Model Architecture: LayoutLLM encodes document images, text, and layout with a document-pretrained model, projects the resulting features, and feeds them with instructions into an LLM.The architecture uses multimodal projectors to align visual and text-layout features with the LLM embedding space.
  • 3.2. Layout-aware Pre-training: Layout-aware pretraining covers document-level, region-level, and segment-level information through tasks such as dense description, layout analysis, and table understanding.These tasks target global document understanding, region identification and classification, and two-dimensional table structure.
  • 3.2.2. Layout-aware Supervised Fine-tuning: LayoutCoT inserts layout-aware intermediate reasoning steps into supervised fine-tuning so the model can focus on question-relevant regions and produce more interpretable answers.The approach adapts chain-of-thought reasoning to document layout and supports interactive inspection and correction.
  • 3.2.2. Layout-aware Supervised Fine-tuning: LayoutCoT data is constructed from HTML, image, and machine-reading-comprehension text datasets by generating document representations, question-answer reasoning, layout reasoning, and document images.The resulting images, question-answer pairs, and LayoutCoTs form the final LayoutCoT dataset.

4. Experiments

Experiments evaluate LayoutLLM in zero-shot document understanding, comparing it with open-source LLMs and MLLMs, testing layout-aware components through ablation, and examining qualitative reasoning and interactive correction. LayoutLLM consistently improves benchmark performance while LayoutCoT supports relevant-area localization, interpretability, and manual correction.

  • Experimental Setup: Zero-shot evaluation uses only benchmark test sets and official image, text, and layout information, covering Document VQA and visual information extraction.The layout-aware pre-training data excludes downstream benchmark training, validation, and test sets.
  • Main Results: Layout Text does not reliably improve over Plain Text: Vicuna-1.5 rises from 48.06% to 59.63% on FUNSD VIE but falls from 66.99% to 56.81% on DocVQA.The authors attribute this instability to difficulty learning formatted layout text and increased token length from coordinates.
  • Main Results: LayoutLLM consistently and significantly outperforms evaluated LLMs and MLLMs across all benchmarks, exceeding trained mPLUG-DocOWL and Qwen-VL by around 10% on DocVQA.It also achieves competitive DocVQA performance against LayoutLMv3, which is fine-tuned on downstream task data.
  • Ablation Study: Layout-aware pre-training and LayoutCoT each improve zero-shot performance, with the ablation showing gains of 1.49% on DocVQA and 3.06% on FUNSD from pre-training.The baseline without either component still reaches 70.82% on DocVQA and 70.96% on FUNSD; LayoutCoT further boosts both document VQA and VIE.
  • Qualitative Results: Qualitative results show LayoutCoT helps the model focus on relevant document areas, use layout information, and expose intermediate location and reasoning information.Without LayoutCoT, the model can select an incorrect column and answer incorrectly even when layout pre-training is used.
  • Interactive Correction: LayoutCoT enables interactive inspection and correction by exposing intermediate results, allowing manually supplied regions to revise incorrect answers.Examples show correction of errors involving the keyword “vendor” and the ambiguous notion of a “second paragraph.”

5. Limitations

LayoutLLM supports interactive correction through LayoutCoT, but remains limited in real-world use and in precise region-level relationship understanding.

  • LayoutLLM cannot yet refuse false-positive outputs or generate hints when answers are absent from documents.The paper identifies these capabilities as crucial for real-world applications but currently absent.
  • Despite improvements from layout-aware pre-training, LayoutLLM struggles to precisely understand region-level relationships.The paper cites Fig. 5(a) as evidence for this limitation.
  • LayoutCoT enables interactive correction, but the authors state that this capability alone is insufficient for real-world applications.

6. Conclusion

The paper proposes LayoutLLM with layout instruction tuning to improve document-layout comprehension and reports effectiveness through extensive experiments.

  • LayoutLLM is proposed as a document-understanding method built around layout instruction tuning.
  • Layout instruction tuning combines layout-aware pre-training with layout-aware supervised fine-tuning.
  • Extensive experiments confirm the effectiveness of LayoutLLM.

A. Dataset Details

The datasets include RVL-CDIP for broad document classification and table-oriented resources containing heterogeneous tables, cell positions, and question-answering pairs.

  • Dataset Details: RVL-CDIP contains 400,000 grayscale document images spanning 16 document classes.Its classes include letters, forms, emails, handwritten documents, advertisements, scientific reports, invoices, presentations, questionnaires, and resumes.
  • Dataset Details: The table data contains heterogeneous tables in both image and HTML formats, with bounding-box positions provided for table cells.
  • Dataset Details: FeTaQA provides 10K Wikipedia-based table question-answering pairs requiring complex reasoning and integration of table information.Its tabular data is extracted from webpages and can be rendered into HTML documents.

A.2. Layout-aware Pre-training Data Construction

Layout-aware pre-training data construction converts document resources and generated or self-supervised targets into instruction-based training examples across document, region, and segment levels.

  • A.2. Layout-aware Pre-training Data Construction: The construction process covers document preprocessing, GPT-3.5 Turbo prompting, and instruction templates for layout-aware pre-training.
  • A.2. Layout-aware Pre-training Data Construction: Document Dense Description uses layout text as input to GPT-3.5 Turbo to generate detailed document descriptions under a length constraint.The process and pretraining templates are illustrated in Fig. 6.
  • A.2. Layout-aware Pre-training Data Construction: Region-level pre-training transforms document-layout and table annotations from original datasets into instruction examples.The corresponding templates are shown in Fig. 8.
  • A.2. Layout-aware Pre-training Data Construction: Segment-level tasks mask text or coordinates, or calculate geometric properties from coordinates, to create self-supervised targets.These targets are then converted into instructions for pre-training.

A.3. Layout-aware SFT Data Construction

Layout-aware SFT data is constructed through document representation, QA and text chain-of-thought generation, LayoutCoT generation, and document image generation. Its document representations combine public image and text documents with GPT-generated HTML documents designed to provide diverse layouts.

  • Layout-aware SFT data construction comprises document representation, QA and Text CoT generation, LayoutCoT generation, and document image generation.
  • The document representations include image documents and text documents from public document-understanding datasets, plus HTML documents freely generated by GPT.
  • GPT-generated HTML documents use randomly sampled LAION-5B images and captions to support diverse document layouts.

B. Training Setup

LayoutLLM initializes its document encoder from LayoutLMv3-large and its language backbone from Vicuna-7B-v1.5 or, in additional experiments, Llama2-7B-chat. Pre-training updates the encoder and projectors with the LLM frozen, whereas SFT fine-tunes the LLM and projectors with the encoder frozen.

  • The document encoder is initialized from LayoutLMv3-large, while the LLM backbone uses Vicuna-7B-v1.5 or Llama2-7B-chat.
  • During pre-training, the LLM remains frozen while the two projectors and document encoder are updated for one epoch.
  • Pre-training uses a 1e-4 learning rate, 0.0001 weight decay, 0.03 warmup ratio, 1.0 maximum gradient norm, and batch size 32 per GPU.
  • During SFT, the LLM and two projectors are fine-tuned while the document encoder remains frozen for three epochs.
  • SFT uses a 2e-5 learning rate, zero weight decay, 0.03 warmup ratio, 1.0 maximum gradient norm, and batch size 8 per GPU.

C. Evaluation Setup

The evaluation setup converts visual information extraction annotations into question-answering tasks and uses deterministic beam-search generation. The supplied construction examples also cover document-level, region-level, segment-level, and layout-aware SFT data generation procedures.

  • Visual Information Extraction: Visual information extraction annotations are transformed into question-answering format for zero-shot evaluation of LLMs and MLLMs.
  • Visual Information Extraction: FUNSD questions ask for values of linking keys, while ambiguous cases involving multiple linked entities are filtered.
  • Visual Information Extraction: CORD and SROIE annotations are converted into entity-type and entity-text pairs, with repeated entity types filtered out.
  • Generation: Testing uses no sampling and applies beam search with beam size 5 across all models.
  • Pre-training Data Construction: Document Dense Description data uses layout text to generate dense descriptions and randomly sampled question templates for document-level pre-training.
  • Pre-training Data Construction: Text and Layout Reconstruction constructs instructions to reconstruct complete document text and layout from document images with OCR or PDF parsing results.
  • Pre-training Data Construction: Document Layout Analysis and Table Understanding instructions are built by transforming layout and table annotations into templates.
  • Pre-training Data Construction: MVLM and Mask Position instructions mask text or position coordinates as inputs and use the masked information as self-supervised targets.
Loading 2404.05225v1…