PaperlayerView exact revision on arXiv

Source-linked AI summary

LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking

Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei

arXiv:2204.08387v3cs.CLcs.CV

TL;DR

Multimodal Document AI models use inconsistent pre-training objectives across text and images, making cross-modal representation learning difficult. LayoutLMv3 unifies masked text and image reconstruction with Word-Patch Alignment in a Transformer that uses linear image patches, and it achieves state-of-the-art results across text-centric and image-centric benchmarks.

  • Problem

    Multimodal Document AI models generally apply masked language modeling to text but use differing image objectives, creating a discrepancy that complicates multimodal representation learning.

  • Method

    LayoutLMv3 jointly encodes text, layout, and raw image patches with unified MLM, MIM, and WPA objectives, without relying on CNN or Faster R-CNN visual backbones.

  • Results

    LayoutLMv3 achieves state-of-the-art performance across text-centric and image-centric Document AI benchmarks, including form understanding, receipt understanding, visual question answering, image classification, and layout analysis.

  • Takeaways & Limitations

    The simple unified architecture and objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks.

Abstract

from arXiv · show

Self-supervised pre-training techniques have achieved remarkable progress in Document AI. Most multimodal pre-trained models use a masked language modeling objective to learn bidirectional representations on the text modality, but they differ in pre-training objectives for the image modality. This discrepancy adds difficulty to multimodal representation learning. In this paper, we propose \textbf{LayoutLMv3} to pre-train multimodal Transformers for Document AI with unified text and image masking. Additionally, LayoutLMv3 is pre-trained with a word-patch alignment objective to learn cross-modal alignment by predicting whether the corresponding image patch of a text word is masked. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks. Experimental results show that LayoutLMv3 achieves state-of-the-art performance not only in text-centric tasks, including form understanding, receipt understanding, and document visual question answering, but also in image-centric tasks such as document image classification and document layout analysis. The code and models are publicly available at \url{https://aka.ms/layoutlmv3}.

1 INTRODUCTION

LayoutLMv3 addresses the mismatch between text and image pre-training objectives in Document AI with unified masking, a simpler visual architecture, and Word-Patch Alignment. It is designed as a general-purpose model and achieves state-of-the-art results across text-centric and image-centric benchmarks.

  • Document AI models commonly use masked language modeling for text but differing image objectives, complicating multimodal representation learning.
  • LayoutLMv3 unifies text and image masking by reconstructing masked word tokens and masked image patch tokens, adding Word-Patch Alignment for cross-modal learning.Image targets are discrete tokens obtained from a discrete VAE, while alignment predicts whether a word’s corresponding image patch is masked.
  • LayoutLMv3 uses raw linear image patches without CNN or Faster R-CNN backbones, reducing parameters and eliminating region annotations.
  • The unified architecture and objectives support both text-centric and image-centric Document AI tasks as a general-purpose pre-trained model.
  • LayoutLMv3 achieves state-of-the-art performance across FUNSD, CORD, DocVQA, RVL-CDIP, and PubLayNet with parameter efficiency.The evaluated tasks cover form understanding, receipt understanding, document visual question answering, document image classification, and document layout analysis.

2 LAYOUTLMV3

LayoutLMv3 uses one Transformer to jointly encode OCR text, layout information, and linearized image patches. Its self-supervised objectives reconstruct masked text and image tokens while learning word–patch alignment.

  • 2.1 Model Architecture: Linear image patches replace CNN or Faster R-CNN visual features, reducing computation, parameters, and dependence on region supervision.
  • 2.1 Model Architecture: Semantic 1D and spatial 2D relative positions are added as bias terms in self-attention for text and image modalities.
  • 2.2 Pre-training Objectives: The combined pre-training loss is L = L_MLM + L_MIM + L_WPA, covering masked language modeling, masked image modeling, and Word-Patch Alignment.
  • 2.2 Pre-training Objectives: MLM predicts masked text tokens from contextualized corrupted text and image sequences while preserving layout information.Text masking uses spans with lengths drawn from a Poisson distribution with λ = 3.
  • 2.2 Pre-training Objectives: MIM predicts discrete image tokens for masked patches, encouraging high-level layout learning rather than noisy low-level detail.
  • 2.2 Pre-training Objectives: WPA learns fine-grained text–image alignment by predicting whether the image patch corresponding to a text word is masked.For the alignment loss, z_l denotes the binary label of the language token at position l.

3 EXPERIMENTS

LayoutLMv3 is evaluated across text-centric and image-centric Document AI benchmarks using a unified Transformer architecture. It achieves strong benchmark results, while ablations show that image embeddings and pre-training objectives are important for stable and effective performance.

  • Fine-tuning on Multimodal Tasks: LayoutLMv3 is fine-tuned on FUNSD, CORD, RVL-CDIP, DocVQA, and PubLayNet across form, receipt, visual question answering, classification, and layout analysis tasks.The experiments cover both text-centric and image-centric document understanding settings.
  • Fine-tuning on Multimodal Tasks: 92.08 F1 on FUNSD is reported for LayoutLMv3LARGE, compared with 85.14 for StructuralLM, although the differing layout-position schemes limit direct comparability.LayoutLMv3 also achieves state-of-the-art F1 scores on CORD for both base and large models.
  • Fine-tuning on Multimodal Tasks: 0.19% and 0.29% absolute accuracy improvements over LayoutLMv2 are reported for LayoutLMv3 base and large models on document image classification.The comparison uses a much simpler linear image embedding instead of a ResNeXt101-FPN embedding.
  • Fine-tuning on Multimodal Tasks: 78.76 ANLS is achieved by LayoutLMv3BASE on DocVQA, improving over LayoutLMv2BASE’s 78.08 with simpler linear image embedding.LayoutLMv3LARGE gains an additional absolute ANLS score of 4.61 over LayoutLMv3BASE.
  • Fine-tuning on a Vision Task: 95.1 overall mAP is achieved on PubLayNet document layout analysis, outperforming the compared ResNet and vision Transformer models across all metrics.The paper attributes the high gain in the Title category to incorporating language modality during pre-training.
  • Ablation Study: Model #2’s loss fails to converge, whereas adding MIM yields normal convergence and adding WPA further decreases the loss.Model #2 adds linearly projected image patches to the text-and-layout baseline; the ablation was fine-tuned on PubLayNet with batch size 16 and learning rate 2e-4.

4 RELATED WORK

Prior multimodal Document AI models commonly apply masked language modeling to text but use differing image objectives, complicating unified multimodal representation learning. Related work spans region, grid, and patch masking alongside coarse- and fine-grained alignment objectives.

  • Document AI architectures: Existing Document AI systems often rely on CNN grid features or object detectors, creating computational bottlenecks or requiring region supervision.
  • Multimodal pre-training: Most multimodal models apply MLM to text but differ in their image pre-training objectives.Image objectives include masked region, grid, and patch modeling.
  • Image masking objectives: Masked region modeling predicts visual features or object labels, while masked grid modeling predicts dictionary indices for masked grid features.
  • Image masking objectives: Patch-level masking methods target visual tokens but may depend on attention weights or architectures that do not support simple linear image embeddings.
  • Cross-modal alignment: Image-text matching provides coarse-grained alignment, whereas word-region and patch-level objectives target finer-grained vision-language correspondence.

5 CONCLUSION AND FUTURE WORK

LayoutLMv3 redesigns LayoutLM with a unified multimodal Transformer architecture and text-image masking objectives. Experiments support its generality across text-centric and image-centric Document AI tasks, while future work targets scaling and few-shot or zero-shot capabilities.

  • Conclusion: LayoutLMv3 redesigns LayoutLM’s architecture and pre-training objectives for multimodal Document AI.
  • Conclusion: Unified MLM, MIM, and word-patch alignment objectives learn multimodal representations without relying on a pre-trained CNN or Faster R-CNN backbone.
  • Conclusion: Experiments demonstrate LayoutLMv3’s generality and superiority on both text-centric and image-centric Document AI tasks.
  • Future work: Future research will investigate larger pre-trained models and few-shot or zero-shot learning for real-world Document AI scenarios.

A.1 LayoutLMv3 in Chinese

The Chinese LayoutLMv3 model is pre-trained on large-scale Chinese document data and evaluated for visual information extraction. On EPHOIE, it achieves a reported SOTA mean F1 score of 99.21%.

  • Pre-training LayoutLMv3 in Chinese: LayoutLMv3-Chinese is a base-size model trained on 50 million Chinese document pages.Its documents are publicly collected and processed following Common Crawl principles.
  • Pre-training LayoutLMv3 in Chinese: The Chinese model initializes its multimodal Transformer encoder and text embeddings from XLM-R, while randomly initializing the remaining parameters.
  • Visual information extraction: EPHOIE contains 1,494 examination-paper images with annotations for 15,771 Chinese text instances and ten character-label categories.
  • Results: 99.21% mean F1: LayoutLMv3-Chinese achieves a reported SOTA result on Chinese visual information extraction.
Loading 2204.08387v3…