PaperlayerView exact revision on arXiv

Source-linked AI summary

LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking

Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei

arXiv:2204.08387v3cs.CLcs.CV

TL;DR

Multimodal Document AI models commonly use masked language modeling for text but differ in their image objectives, complicating cross-modal alignment. LayoutLMv3 combines unified text and image masking with word-patch alignment, and achieves state-of-the-art performance across text-centric and image-centric benchmarks.

  • Problem

    Multimodal Document AI models use masked language modeling for text but differ in image pre-training objectives, which adds difficulty to cross-modal alignment learning.

    Source: §1 · pp. 1–2

  • Method

    LayoutLMv3 uses unified masked language and image modeling objectives and predicts whether each word’s corresponding image patch is masked to learn cross-modal alignment.

    Source: §1 · p. 2

  • Results

    LayoutLMv3 achieves state-of-the-art performance across five public benchmarks spanning text-centric and image-centric Document AI tasks, with parameter efficiency.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results demonstrate LayoutLMv3’s generality for both text-centric and image-centric Document AI tasks.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

Self-supervised pre-training techniques have achieved remarkable progress in Document AI. Most multimodal pre-trained models use a masked language modeling objective to learn bidirectional representations on the text modality, but they differ in pre-training objectives for the image modality. This discrepancy adds difficulty to multimodal representation learning. In this paper, we propose \textbf{LayoutLMv3} to pre-train multimodal Transformers for Document AI with unified text and image masking. Additionally, LayoutLMv3 is pre-trained with a word-patch alignment objective to learn cross-modal alignment by predicting whether the corresponding image patch of a text word is masked. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks. Experimental results show that LayoutLMv3 achieves state-of-the-art performance not only in text-centric tasks, including form understanding, receipt understanding, and document visual question answering, but also in image-centric tasks such as document image classification and document layout analysis. The code and models are publicly available at \url{https://aka.ms/layoutlmv3}.

1 INTRODUCTION

LayoutLMv3 addresses the mismatch between text and image pre-training by combining unified masking with word-patch alignment in a Transformer for Document AI.

1 INTRODUCTION

  • LayoutLMv3 uses unified masked language modeling and masked image modeling, plus Word-Patch Alignment to predict whether a word’s corresponding image patch is masked.Masked image targets are discrete patch tokens derived from latent codes of a discrete VAE.Source: §1 · p. 2
  • LayoutLMv3 uses raw image patches without complex preprocessing or CNN image embeddings, saving parameters and eliminating region annotations.It is described as the first multimodal Document AI model without CNNs for image embeddings or a pre-trained CNN or Faster R-CNN backbone for visual features.Source: §1 · p. 2
  • The unified architecture and objectives make LayoutLMv3 a general-purpose model for both text-centric and image-centric Document AI tasks.The authors demonstrate multimodal Transformers’ generality on vision tasks in Document AI and state that the code and models are publicly available.Source: §1 · p. 2
  • LayoutLMv3 achieves state-of-the-art, parameter-efficient performance across five benchmarks spanning form, receipt, document-VQA, document-image classification, and layout analysis.The benchmarks are FUNSD, CORD, DocVQA, RVL-CDIP, and PubLayNet, respectively.Source: §1 · p. 2

2 LAYOUTLMV3

LayoutLMv3 combines text and image inputs in a unified multimodal Transformer and pre-trains it with masked language modeling, masked image modeling, and word-patch alignment. Its architecture uses OCR text with layout positions and linearly projected image patches, while its objectives train contextual reconstruction and cross-modal alignment.

2.1 Model Architecture

  • Text inputs combine OCR-extracted tokens, RoBERTa-initialized word embeddings [36], token positions, and 2D layout positions, while text and image sequences are jointly contextualized by one Transformer.The Transformer concatenates the text and image embedding sequences and produces contextual representations for both modalities.Source: §2.1 · p. 2
  • Words in a segment share normalized 2D layout coordinates, while image inputs are linearly projected uniform patches with learned 1D positions, avoiding CNN feature extraction.The paper identifies LayoutLMv3 as the first Document AI multimodal model that does not rely on CNNs for image features.Source: §2.1 · p. 3
  • Self-attention also uses semantic 1D relative-position and spatial 2D relative-position biases for text and image modalities, following LayoutLMv2 [56].Source: §2.1 · p. 3

2.2 Pre-training Objectives

  • MLM masks 30% of text tokens in spans with lengths drawn from a Poisson distribution (λ = 3) and predicts the masked tokens from contextual representations.Layout positions remain unchanged, helping the objective teach correspondence between layout information and text and image context.Source: §2.2 · pp. 3–4
  • MIM randomly masks about 40% of image tokens blockwise and reconstructs discrete image-token labels, encouraging learning of high-level layout structures rather than noisy low-level details.The image-token labels are produced by an image tokenizer.Source: §2.2 · p. 4
  • WPA predicts whether the image patches corresponding to each unmasked text token are masked, using aligned or unaligned labels and a two-layer MLP with binary cross-entropy.Masked text tokens are excluded from the WPA loss to prevent learning correspondence between masked words and image patches.Source: §2.2 · p. 4

3 EXPERIMENTS

LayoutLMv3 is evaluated on multimodal and vision-only document tasks, with results showing strong performance across both. The experiments also examine how image embeddings and pre-training objectives affect performance.

3.1 Model Configurations

  • LayoutLMv3 uses base and large Transformer configurations, with maximum text length 512 and 196 image patches from 224 × 224 inputs.Base uses 12 layers, 12 attention heads, and hidden size 768; large uses 24 layers, 16 heads, and hidden size 1,024.Source: §3.1 · p. 4

3.2 Pre-training LayoutLMv3

  • LayoutLMv3 is pre-trained on 11 million IIT-CDIP images for 500,000 steps, initialized with RoBERTa text weights and a DiT image tokenizer.Training uses batch size 2,048; base and large use learning rates 1e−4 and 5e−5, respectively.Source: §3.2 · p. 4

3.3 Fine-tuning on Multimodal Tasks

  • LayoutLMv3 reports leading results across form and receipt understanding, document image classification, and DocVQA.The large model scores 92.08 F1 on FUNSD versus StructuralLM’s 85.14, though their segment-level positions make the scores not directly comparable; both model sizes achieve SOTA on CORD. On RVL-CDIP, LayoutLMv3 improves over LayoutLMv2 by 0.19% (base) and 0.29% (large); on DocVQA, base improves ANLS from 78.08 to 78.76, and large gains 4.61 ANLS points over base.Source: Table 1 · p. 5, §3.3 · p. 6

3.4 Fine-tuning on a Vision Task

  • LayoutLMv3 achieves 95.1 overall mAP on PubLayNet, outperforming the compared backbones across all metrics.Its high gain in the Title category is attributed by the authors to incorporating language modality during pre-training.Source: Table 2 · p. 6, §3.4 · p. 7

3.5 Ablation Study

  • Adding linear image embeddings alone can hurt CORD and RVL-CDIP performance and cause PubLayNet loss to diverge, while MIM improves those tasks and enables convergence.The authors report that linear image embeddings can improve FUNSD without MIM, and conclude MIM is critical for vision tasks such as PubLayNet layout analysis.Source: §3.5 · p. 7
  • WPA consistently improves all evaluated tasks over MIM alone and further decreases PubLayNet fine-tuning loss.The authors interpret this as evidence that WPA benefits both cross-modal and image representation learning.Source: §3.5 · p. 7, Figure 4 · p. 7
  • The image patch projection adds 0.6M parameters, while the MIM and WPA heads add 6.9M and 0.6M parameters during pre-training.The paper describes the image-embedding overhead as marginal relative to the 39.2M-parameter MLM head and notes that the standalone image tokenizer is excluded.Source: §3.5 · p. 8

4 RELATED WORK

LayoutLMv3 builds on prior document and vision-language pre-training while addressing limitations in visual encoding, image masking, and cross-modal alignment. The related work frames its contributions as CNN-free image embeddings, effective masked image modeling for linear patch embeddings, and a unified masking-based alignment objective.

4 RELATED WORK

  • LayoutLMv3 is presented as the first Document AI multimodal model to use image embeddings without CNNs, following ViLT’s lightweight visual-embedding approach [22].Earlier document models combined CNNs with Transformers, using grid features or detector-derived region features that could entail heavy computation or require region supervision [2] [31] [40] [49] [54] [56].Source: §4 · p. 8
  • Reconstructive pre-training spans BERT’s masked language modeling and vision methods that predict masked image content, including ViT, BEiT, and document-focused DiT [9] [11] [30].ViT reconstructs masked-patch mean colors, while BEiT predicts discrete visual tokens; DiT extends BEiT to document-image layout analysis.Source: §4 · p. 8
  • LayoutLMv3 demonstrates the effectiveness of masked image modeling for linear patch embeddings, unlike prior ViLT and METER patch-masking approaches that degraded downstream performance [12] [22].Prior multimodal MIM objectives included masked region modeling, masked grid modeling, and masked patch modeling, with different reconstruction targets [5] [19] [31] [37] [48].Source: §4 · p. 8
  • Prior cross-modal alignment ranges from image-text matching and word-region or patch-level alignment to document-specific methods; LayoutLMv3 uses MIM masking to form aligned and unaligned pairs [2] [5] [19] [22] [56].UDoc aligns text and image from the same area, while LayoutLMv2 predicts whether text tokens are covered in raw images.Source: §4 · p. 8

5 CONCLUSION AND FUTURE WORK

LayoutLMv3 redesigns the architecture and pre-training objectives for Document AI, using unified text and image masking without a pre-trained CNN or Faster R-CNN backbone. Experiments demonstrate its generality and superiority across text-centric and image-centric tasks, while future work will investigate scaling and few-shot and zero-shot learning.

5 CONCLUSION AND FUTURE WORK

  • LayoutLMv3 demonstrates generality and superiority on both text-centric and image-centric Document AI tasks.The conclusion attributes these results to its simple architecture and unified objectives.Source: §5 · p. 8
  • LayoutLMv3 avoids pre-trained CNN or Faster R-CNN visual-feature backbones, saving parameters and eliminating region annotations.Source: §5 · p. 8
  • Its unified pre-training objectives are masked language modeling, masked image modeling, and word-patch alignment.Source: §5 · p. 8
  • Future work will investigate scaling up pre-trained models to use more training data and explore few-shot and zero-shot learning for real-world business scenarios.Source: §5 · p. 8

A APPENDIX · A.1 LayoutLMv3 in Chinese

LayoutLMv3-Chinese was pretrained on 50 million Chinese document pages and achieved a SOTA mean F1 of 99.21% on Chinese visual information extraction. The appendix describes its initialization, task formulation, and EPHOIE evaluation setup.

A.1 LayoutLMv3 in Chinese

  • The base-size LayoutLMv3-Chinese model is pretrained on 50 million Chinese document pages collected from public digital-born documents.The collection process follows Common Crawl principles.Source: §A.1 · p. 9
  • The multimodal encoder and text embedding layer are initialized from XLM-R [7], while the remaining model parameters are randomly initialized.Other training settings are the same as LayoutLMv3.Source: §A.1 · p. 9
  • Visual information extraction is treated as sequence labeling, predicting each text token’s label from its last hidden state using a linear layer.The task aims to tag each word with a predefined label.Source: §A.1 · p. 9
  • EPHOIE [51] contains 1,494 examination-paper-head images with diverse layouts and backgrounds, annotated for 15,771 Chinese text instances.The evaluation assigns each character one of ten entity categories.Source: §A.1 · p. 9
  • The EPHOIE [51] training and test sets contain 1,183 and 311 images, respectively.LayoutLMv3-Chinese is fine-tuned for 100 epochs with batch size 16 and learning rate 5e−5, using linear warmup over the first epoch.Source: §A.1 · p. 9
  • LayoutLMv3-Chinese achieves a SOTA mean F1 score of 99.21% on Chinese visual information extraction and performs better on most reported metrics.The authors report that it significantly benefits the VIE task in Chinese.Source: §A.1 · p. 9

Related papers

Loading 2204.08387v3…