Source-linked AI summary
LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou
TL;DR
Visually-rich document understanding needs unified modeling of text, layout, and visual information rather than relying on text alone or shallow multimodal fusion. LayoutLMv2 addresses this with multimodal pre-training, new cross-modal objectives, and spatial-aware attention, achieving state-of-the-art results across six downstream tasks. The authors identify multilingual expansion as future work.
Problem
Visually-rich document understanding requires textual, visual, and layout information to be jointly modeled because relevant fields depend on document position, style, format, and content.
Method
LayoutLMv2 integrates text, layout, and image information in one multimodal Transformer during pre-training, adding spatial-aware attention plus text-image alignment and matching objectives.
Results
LayoutLMv2 substantially outperforms strong baselines and achieves new state-of-the-art results across six visually-rich document understanding tasks.
Takeaways & Limitations
The results support multimodal pre-training as a promising approach for visually-rich document understanding across conventional tasks and document-image question answering.
Takeaways & Limitations
The paper leaves multilingual expansion, especially for non-English areas, as future work.
Abstract
from arXiv · showhide
Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. We propose LayoutLMv2 architecture with new pre-training tasks to model the interaction among text, layout, and image in a single multi-modal framework. Specifically, with a two-stream multi-modal Transformer encoder, LayoutLMv2 uses not only the existing masked visual-language modeling task but also the new text-image alignment and text-image matching tasks, which make it better capture the cross-modality interaction in the pre-training stage. Meanwhile, it also integrates a spatial-aware self-attention mechanism into the Transformer architecture so that the model can fully understand the relative positional relationship among different text blocks. Experiment results show that LayoutLMv2 outperforms LayoutLM by a large margin and achieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks, including FUNSD (0.7895 $\to$ 0.8420), CORD (0.9493 $\to$ 0.9601), SROIE (0.9524 $\to$ 0.9781), Kleister-NDA (0.8340 $\to$ 0.8520), RVL-CDIP (0.9443 $\to$ 0.9564), and DocVQA (0.7295 $\to$ 0.8672). We made our model and code publicly available at \url{https://aka.ms/layoutlmv2}.
1 Introduction
Visually-rich document understanding requires jointly modeling textual, visual, and layout information, motivating LayoutLMv2’s unified multimodal pre-training framework. The model adds cross-modal objectives and spatial-aware attention, achieving state-of-the-art results across six downstream tasks.
- Visually-rich documents require jointly modeling text, visual information, and layout because relevant fields depend on position, style, format, and content.
- The model introduces spatial-aware self-attention with 2-D relative position representations to model contextual spatial relationships among document elements.
- LayoutLMv2 achieves new state-of-the-art results across six tasks, including form, receipt, long-document, document-image classification, and document-image question answering.The evaluated datasets are FUNSD, CORD, SROIE, Kleister-NDA, RVL-CDIP, and DocVQA.
- LayoutLMv2 integrates document text, layout, and visual information during pre-training to learn cross-modal interactions end-to-end.
- LayoutLMv2 adds text-image alignment and text-image matching to masked visual-language modeling, enforcing alignment among modalities during pre-training.
2 Approach
LayoutLMv2 builds a unified multimodal Transformer that integrates text, visual, and layout information during pre-training. Its spatial-aware attention and cross-modal pre-training tasks model document structure and text–image correspondence.
- Model Architecture: LayoutLMv2 uses a multimodal Transformer backbone that takes text, visual, and layout information as input.Visual information is integrated during pre-training rather than only during fine-tuning.
- Text Embedding: The model tokenizes OCR text and combines token, one-dimensional positional, and segment embeddings for each text token.The text sequence is padded to a fixed maximum length.
- Visual Embedding: A CNN-based visual encoder converts the page image into a fixed-length visual-token sequence whose dimensionality is aligned with text embeddings.The visual backbone is ResNeXt-FPN, and its parameters can be updated through backpropagation.
- Layout Embedding: Layout embeddings encode bounding-box coordinates, widths, and heights to represent the spatial layout of OCR and visual tokens.Coordinates are normalized and discretized, while special tokens receive an empty bounding box.
- Multimodal Encoder: The encoder concatenates visual and text embeddings, adds layout embeddings, and uses spatial-aware self-attention to model relative semantic and spatial positions.Learnable one-dimensional and two-dimensional relative-position biases are shared across encoder layers but differ among attention heads.
- Pre-training Tasks: Pre-training combines masked visual-language modeling with text-image alignment and text-image matching to learn cross-modal correspondence.MVLM recovers masked text while retaining layout; TIA predicts whether text lines correspond to covered image regions; TIM distinguishes matching from mismatched or missing page images.
3 Experiments
Experiments evaluate LayoutLMv2 across six visually-rich document understanding datasets using standard pre-training and fine-tuning settings. LayoutLMv2 consistently outperforms strong baselines, while ablations attribute gains to visual information, cross-modality tasks, spatial-aware attention, and initialization.
- 3.1 Data: Six downstream datasets cover entity extraction, document image classification, and visual question answering in visually-rich document understanding.The datasets are FUNSD, CORD, SROIE, Kleister-NDA, RVL-CDIP, and DocVQA.
- 3.2 Settings: LayoutLMv2 is evaluated against BERT, UniLMv2, and vanilla LayoutLM across all experiment settings.The baselines include text-only pre-trained models and the previous LayoutLM model.
- 3.3 Results: LayoutLMv2 achieves new state-of-the-art results on all six downstream tasks and significantly outperforms strong baselines including vanilla LayoutLM.The entity extraction results use entity-level F1 scores, while RVL-CDIP uses classification accuracy and DocVQA uses ANLS.
- 3.3 Results: 95.64% accuracy is achieved by LayoutLMv2LARGE on RVL-CDIP, improving classification accuracy by more than 1.2 percentage points over previous state-of-the-art results.The result supports benefits from combining textual and visual information for document image classification.
- 3.3 Results: More than 1.6% ANLS improvement is obtained by successively fine-tuning LayoutLMv2LARGE on question generation and DocVQA data, achieving new state-of-the-art performance.The single model is fine-tuned using the question generation dataset followed by the DocVQA dataset.
- 3.4 Ablation Studies: Ablations show that visual information, TIA, TIM, SASAM, and UniLMv2 initialization each contribute to performance, with TIA benefiting more than TIM.Using TIA and TIM together is more effective than using either alone, and SASAM further improves accuracy.
4 Related Work
Prior VrDU work leveraged pre-training in NLP and CV, including multi-modal vision-language models and LayoutLM, to combine textual, visual, and layout information. LayoutLMv2 extends this line with a unified multi-modal pre-training framework.
- Pre-training techniques from NLP and CV have increasingly been leveraged for visually-rich document understanding.
- ViLBERT introduced a two-stream multi-modal model for learning joint representations of image content and natural language.
- VL-BERT extended the Transformer backbone to accept visual and linguistic embedded features as input.
- LayoutLM jointly modeled interactions between text and layout information across scanned document images for document understanding tasks.
- LayoutLMv2 builds on LayoutLM by combining textual, layout, and visual information in a single multi-modal pre-training framework.
5 Conclusion
LayoutLMv2 is a multi-modal pre-training approach that integrates text, layout, and image information while modeling relationships among bounding boxes. Across six VrDU tasks, it substantially outperformed state-of-the-art baselines; future work includes architectural, pre-training, and multilingual extensions.
- LayoutLMv2 integrates text, layout, and image information during pre-training in a single multi-modal framework.
- Its spatial-aware self-attention mechanism captures relative relationships among different bounding boxes.
- Experiments on six VrDU tasks showed that pre-trained LayoutLMv2 substantially outperformed state-of-the-art baselines in document intelligence.
- Future research will explore LayoutLM architecture and pre-training strategies and extend LayoutLMv2 to additional languages, especially non-English areas.
A Details of Datasets
The evaluation and pre-training data span multiple visually-rich document understanding settings, including forms, receipts, long agreements, document images, and visual question answering. LayoutLMv2 is pre-trained on over 11 million scanned document pages and evaluated across six benchmark tasks.
- Pre-training Dataset: LayoutLMv2 is pre-trained on the IIT-CDIP Test Collection, which contains over 11 million scanned document pages.
- FUNSD: FUNSD evaluates semantic entity labeling on 199 noisy scanned forms, using entity-level F1 as its metric.
- CORD: CORD provides 800 training, 100 validation, and 100 test receipts for receipt key information extraction using photos and OCR annotations.
- SROIE: SROIE extracts values for up to four predefined receipt keys from 626 training and 347 test samples, evaluated with entity-level F1.
- Kleister-NDA: Kleister-NDA contains 254 training, 83 validation, and 203 test non-disclosure agreements for extracting four fixed keys.
- RVL-CDIP: RVL-CDIP defines 16-class document image classification over 400,000 grayscale images, evaluated by overall classification accuracy.
- DocVQA: DocVQA contains 50,000 questions over more than 12,000 document pages, organized as page image, question, and answer triples.
B Model Training Details
LayoutLMv2 uses task-specific fine-tuning heads for visual question answering, document image classification, and sequential labeling. Pre-training uses Adam with specified regularization and schedules, with different batch sizes and epoch counts for BASE and LARGE models.
- Pre-training: LayoutLMv2 pre-training uses Adam with learning rate 2 × 10^-5, weight decay 1 × 10^-2, and linearly warmed-up then decayed learning rates.
- Pre-training: LayoutLMv2BASE is trained with batch size 64 for 5 epochs, while LayoutLMv2LARGE uses batch size 2048 for 20 epochs on IIT-CDIP.
- Fine-tuning for Visual Question Answering: For visual question answering, a token-level classifier is built on the text output representations and uses a maximum sequence length of L = 384.
- Fine-tuning for Document Image Classification: For document image classification, pooled visual features before and after encoding are concatenated with the [CLS] feature before the final classification layer.
- Fine-tuning for Sequential Labeling: FUNSD, SROIE, CORD, and Kleister-NDA are fine-tuned as sequential labeling tasks with a token-level classifier predicting BIO tags.
C Detailed Experiment Results
The detailed results organize entity-extraction evaluation across FUNSD, CORD, SROIE, and Kleister-NDA using entity-level metrics, with separate tables for each dataset.
- Entity Extraction Tasks: Entity-extraction results are reported separately for FUNSD, CORD, SROIE, and Kleister-NDA.Tables 6–9 provide per-task results for the four datasets.
- Evaluation Metrics: The reported metrics include Precision, Recall, and F1 for the entity-extraction evaluations.The tables use entity-level evaluation, with F1 identified as the task-level score.
- FUNSD: Table 6 reports model accuracy on FUNSD using entity-level Precision, Recall, and F1.
- CORD: Table 7 reports model accuracy on CORD using entity-level Precision, Recall, and F1.
- SROIE: Table 8 reports model accuracy on SROIE using entity-level Precision, Recall, and F1.
- Kleister-NDA: Table 9 reports entity-level F1 on the Kleister-NDA validation set using the official evaluation toolkit.