Source-linked AI summary
LayoutLM: Pre-training of Text and Layout for Document Image Understanding
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
TL;DR
Document AI models often underuse unlabeled scanned documents and omit joint modeling of text, layout, and visual information, despite the diversity of business-document formats. LayoutLM addresses this gap by combining token, 2-D position, and image embeddings with self-supervised pre-training on large-scale scanned documents. It substantially outperforms several state-of-the-art pre-trained models across form understanding, receipt understanding, and document image classification.
Problem
Existing document AI approaches rely on limited labeled data and usually pre-train computer-vision or language models separately rather than jointly modeling text and layout.
Method
LayoutLM extends BERT with token, 2-D position, and image embeddings, using masked visual-language modeling and multi-label document classification on large-scale scanned documents.
Results
LayoutLM substantially outperforms several state-of-the-art pre-trained models across form understanding, receipt understanding, and scanned document image classification.
Takeaways & Limitations
Joint pre-training of text and layout in a single framework achieves strong performance across multiple document image understanding tasks.
Takeaways & Limitations
Future work includes more data and computation, a LARGE architecture, image embeddings during pre-training, and alternative architectures or self-supervised objectives.
Abstract
from arXiv · showhide
Pre-training techniques have been verified successfully in a variety of NLP tasks in recent years. Despite the widespread use of pre-training models for NLP applications, they almost exclusively focus on text-level manipulation, while neglecting layout and style information that is vital for document image understanding. In this paper, we propose the \textbf{LayoutLM} to jointly model interactions between text and layout information across scanned document images, which is beneficial for a great number of real-world document image understanding tasks such as information extraction from scanned documents. Furthermore, we also leverage image features to incorporate words' visual information into LayoutLM. To the best of our knowledge, this is the first time that text and layout are jointly learned in a single framework for document-level pre-training. It achieves new state-of-the-art results in several downstream tasks, including form understanding (from 70.72 to 79.27), receipt understanding (from 94.02 to 95.24) and document image classification (from 93.07 to 94.42). The code and pre-trained LayoutLM models are publicly available at \url{https://aka.ms/layoutlm}.
1 INTRODUCTION
Document AI must handle business documents whose layouts, formats, and scan quality vary widely, while existing approaches underuse unlabeled data and do not jointly pre-train text and layout. LayoutLM addresses this gap with multimodal pre-training and shows strong results across three document understanding tasks.
- Motivation: Business documents vary across plain text, multi-column layouts, tables, forms, and figures, making understanding challenging.Scanned-image quality and complex template structures add further difficulty.
- Motivation: Existing document AI methods rely on limited labeled data and typically pre-train either computer-vision or natural-language models without jointly modeling text and layout.The paper identifies self-supervised pre-training of text and layout as an important research direction.
- Evaluation: LayoutLM is evaluated on form understanding, receipt understanding, and scanned document image classification using three benchmark datasets.The benchmarks are FUNSD, SROIE, and RVL-CDIP.
- Results: The experiments report that pre-trained LayoutLM substantially outperforms several state-of-the-art pre-trained models across these document understanding tasks.The paper presents this result as evidence of the advantage of pre-training text and layout information.
- Contribution: LayoutLM jointly pre-trains textual and layout information from scanned document images in a single framework, while also leveraging image features.Its objectives include masked visual-language modeling and multi-label document classification.
2 LAYOUTLM
LayoutLM extends BERT for visually rich documents by combining text, two-dimensional position, and image embeddings. Its pre-training uses masked visual-language modeling and multi-label document classification before fine-tuning on document understanding tasks.
- 2.1 The BERT Model: BERT uses a bidirectional Transformer with word, sequence-position, and segment embeddings, followed by pre-training and task-specific fine-tuning.Its pre-training objectives are masked language modeling and next sentence prediction.
- 2.2 The LayoutLM Model: BERT-like models mainly use text, whereas visually rich documents also contain layout and style information that can enrich document representations.The paper motivates aligning document layouts with input text.
- 2.2 The LayoutLM Model: LayoutLM adds 2-D position and image embeddings to BERT alongside token embeddings to represent spatial relationships and visual appearance.The 2-D position embedding models relative document position, while image embeddings represent OCR-aligned word regions from Faster R-CNN.
- Pre-training Objectives: Masked Visual-Language Modeling masks tokens while retaining their 2-D positions, training the model to recover tokens from textual and spatial context.This objective extends masked language modeling with layout clues.
- Pre-training Objectives: Multi-label Document Classification uses document tags during pre-training to learn document-level representations across domains.The objective is motivated by documents carrying multiple tags in the IIT-CDIP collection.
- Downstream Tasks: The pre-trained model is fine-tuned for form understanding, receipt understanding, and document image classification.It uses token-level sequential labeling for the first two tasks and the [CLS] representation for classification.
3 EXPERIMENTS
LayoutLM is pretrained on over 11 million scanned document images and evaluated across form understanding, receipt extraction, and document image classification. Across these experiments, combining text, layout, and image information achieves strong results, including improved FUNSD performance in low-resource settings.
- 3.1 Pre-training Dataset: Over 11 million scanned document images from IIT-CDIP provide the large-scale pretraining corpus, alongside OCR text and document metadata.The dataset contains more than 6 million documents, with scanned images and XML-stored text and metadata.
- 3.2 Fine-tuning Dataset: FUNSD contains 199 annotated scanned forms with 9,707 semantic entities and 31,485 words for form-understanding evaluation.Entities include labels, bounding boxes, links, and word lists.
- 3.2 Fine-tuning Dataset: SROIE contains 626 training and 347 testing receipts labeled for company, date, address, and total extraction.Evaluation uses exact-match entity-recognition F1.
- 3.2 Fine-tuning Dataset: RVL-CDIP contains 400,000 grayscale images across 16 document classes, evaluated using overall classification accuracy.The dataset provides 320,000 training, 40,000 validation, and 40,000 test images.
- 3.6 Results: LayoutLM obtains 0.7927 F1 on FUNSD when text, layout, and image information are used together, surpassing BERT and RoBERTa baselines.With the BASE architecture and 11M training data, LayoutLM reaches 0.7866 F1 before adding the multi-label document classification loss and image information.
- 3.6 Results: FUNSD accuracy increases with more pretraining data and epochs, supporting the effectiveness of text-and-layout pretraining when only 149 images are available for fine-tuning.The reported trend is especially relevant to low-resource scanned-document understanding.
4 RELATED WORK
Document analysis research progressed from rule-based and conventional machine-learning methods to deep learning, but existing approaches still faced limited labeled data and separate treatment of text and layout.
- Rule-based Approaches: Rule-based document analysis uses bottom-up component grouping or top-down recursive page splitting to construct document structure.These methods can perform well on some documents but require extensive human effort to design rules.
- Conventional Machine Learning Approaches: Conventional machine-learning approaches formulate document layout analysis as parsing and learn features and parameters during the parsing process.
- Deep Learning Approaches: Deep-learning approaches include pixel-level semantic extraction, multimodal document modeling, segmentation-mask prediction, and localized region detection.
- Deep Learning Approaches: Existing deep-learning methods rely on limited labeled data and typically use either pre-trained CV or NLP models rather than jointly pre-training text and layout.LayoutLM addresses both limitations and reports better performance than previous baselines.
5 CONCLUSION AND FUTURE WORK
The paper presents LayoutLM as a single-framework pre-training technique combining text, layout, and image information for document understanding. It evaluates the model across three tasks and identifies expanded scale, architectures, and objectives as future directions.
- Conclusion: LayoutLM combines token, layout, and image embeddings in a Transformer-based framework that can be trained self-supervised on large-scale unlabeled scanned documents.
- Conclusion: LayoutLM is evaluated on form understanding, receipt understanding, and scanned document image classification.
- Conclusion: Experiments show that LayoutLM substantially outperforms several state-of-the-art pre-trained models across the evaluated tasks.
- Future Work: Future work includes using more data and computation, training the LARGE architecture with text and layout, adding image embeddings during pre-training, and exploring new architectures and objectives.