Source-linked AI summary
LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
TL;DR
Visually-rich document understanding needs multilingual multimodal models because non-English documents are poorly served by translation and existing multilingual text models do not incorporate document layout and images. LayoutXLM extends multimodal pre-training to multilingual documents and introduces XFUND for seven-language evaluation; it significantly outperforms state-of-the-art multilingual baselines. The paper makes both resources publicly available, while noting that broader language, layout, and template coverage remains future work.
Problem
Multilingual visually-rich document understanding requires models that jointly handle text, layout, and images rather than relying on unsatisfactory translation of document images.
Method
LayoutXLM extends LayoutLMv2 with multimodal pre-training on large-scale multilingual real-world documents, evaluated with the seven-language XFUND benchmark.
Results
LayoutXLM significantly outperformed state-of-the-art multilingual baselines for multilingual document understanding on XFUND and related transfer settings.
Takeaways & Limitations
LayoutXLM and XFUND provide publicly available resources for multilingual visually-rich document understanding research.
Takeaways & Limitations
The authors identify the need to expand multilingual training data to cover more languages, document layouts, and templates.
Abstract
from arXiv · showhide
Multimodal pre-training with text, layout, and image has achieved SOTA performance for visually-rich document understanding tasks recently, which demonstrates the great potential for joint learning across different modalities. In this paper, we present LayoutXLM, a multimodal pre-trained model for multilingual document understanding, which aims to bridge the language barriers for visually-rich document understanding. To accurately evaluate LayoutXLM, we also introduce a multilingual form understanding benchmark dataset named XFUND, which includes form understanding samples in 7 languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese), and key-value pairs are manually labeled for each language. Experiment results show that the LayoutXLM model has significantly outperformed the existing SOTA cross-lingual pre-trained models on the XFUND dataset. The pre-trained LayoutXLM model and the XFUND dataset are publicly available at https://aka.ms/layoutxlm.
1 Introduction
LayoutXLM addresses multilingual visually-rich document understanding by extending multimodal pre-training to multilingual real-world documents. The paper also introduces XFUND, a seven-language form-understanding benchmark, and reports superior performance over multilingual baselines.
- Motivation: Multimodal pre-training jointly learns text, layout, and image information for visually-rich document understanding.This approach has achieved strong performance on several document understanding benchmarks.
- Motivation: Nearly 40% of digital documents on the web are in non-English languages, while machine translation can provide poor quality on document images.The paper therefore motivates multilingual pre-training on real document data.
- Approach: LayoutXLM extends LayoutLMv2 into a multimodal pre-trained model for multilingual visually-rich document understanding.It retains the LayoutLMv2 architecture and is initialized with InfoXLM.
- Benchmark: The paper introduces XFUND, a multilingual form-understanding benchmark with human-labeled key-value pairs in seven languages.The languages are Chinese, Japanese, Spanish, French, Italian, German, and Portuguese.
- Results: LayoutXLM outperformed existing state-of-the-art multilingual baseline models on XFUND.The authors make both the pre-trained model and XFUND publicly available.
2 Approach
The approach uses a multimodal Transformer that combines text, layout, and image representations with cross-modal pre-training objectives. LayoutXLM is trained on large multilingual document collections assembled from digital-born PDFs and scanned documents.
- Model architecture: LayoutXLM follows LayoutLMv2 and uses a multimodal Transformer for multilingual document understanding.The model transfers the architecture to large-scale multilingual document datasets.
- Model architecture: The model combines text, layout, and image embeddings before encoding them with spatial-aware self-attention.Its contextual representations feed task-specific layers.
- Pre-training objectives: The pre-training framework includes Multilingual Masked Visual-Language Modeling, Text-Image Alignment, and Text-Image Matching.These objectives model text-layout alignment, fine-grained text-image alignment, and whether text and images come from the same page.
- Pre-training objectives: LayoutXLM uses character-level bounding boxes to avoid language-specific preprocessing differences in linguistic units.This replaces word-level layout assignment used in LayoutLM and LayoutLMv2.
- Pre-training data: The pre-training corpus covers 53 languages and combines publicly available multilingual digital-born PDFs with scanned documents.The sampling procedure uses 22 million visually rich multilingual documents and 8 million scanned English documents, totaling 30 million.
3 XFUND: A Multilingual Form Understanding Benchmark
XFUND is a multilingual form-understanding benchmark extending FUNSD to seven languages, with manually labeled key-value structures and evaluation tasks for semantic entities and relations.
- 3 XFUND: A Multilingual Form Understanding Benchmark: XFUND extends the FUNSD benchmark to Chinese, Japanese, Spanish, French, Italian, German, and Portuguese forms.
- 3.1 Task description: Key-value extraction is evaluated through semantic entity recognition and relation extraction.Semantic entity recognition extracts and classifies entities, while relation extraction predicts relations between semantic entities.
- 3.1 Task description: Semantic entity recognition maps document tokens and entity-label sets to predicted semantic entities.Each token includes text and bounding-box coordinates, preserving document layout information.
- 3.1 Task description: Relation extraction predicts labeled relations between pairs of semantic entities, with this work focusing on key-value relations.
- 3.1 Task description: Sampled Chinese and Italian forms visualize headers in red, keys in green, and values in blue.
- 3.2 Data Collection and Labeling: XFUND forms are built from public templates, filled with synthetic information, scanned into document images, and processed for OCR and key-value labeling.
- 3.2 Data Collection and Labeling: Human annotators group OCR tokens into entities, assign predefined labels, and link related entities into key-value pairs.
- 3.3 Dataset Statistics: The benchmark contains 1,393 fully annotated forms: 199 per language, with 149 training forms and 50 test forms per language.
4 Experiments
The experiments evaluate LayoutXLM on XFUND through language-specific, zero-shot cross-lingual, and multilingual multitask fine-tuning across semantic entity recognition and relation extraction.
- 4 Experiments: Language-specific fine-tuning compares LayoutXLM with XLM-R and InfoXLM when training and testing use the same language.The experiments report F1 accuracy for both XFUND subtasks.
- 4 Experiments: XFUND evaluates models on multilingual forms using semantic entity recognition and relation extraction.The benchmark includes human-labeled forms and key-value annotations across seven languages.
- 4.2 Results: LayoutXLM LARGE achieves the highest F1 scores in both SER and RE under language-specific fine-tuning.The reported improvement over XLM-R and InfoXLM supports transfer of pre-trained knowledge to downstream tasks.
- 4.2 Results: In zero-shot transfer, models are fine-tuned on English FUNSD and evaluated on target languages, with LayoutXLM significantly outperforming text-based models.The result is attributed to capturing common layout invariance across languages for form understanding.
- 4.2 Results: Multitask fine-tuning trains on all eight languages simultaneously and further improves performance compared with language-specific fine-tuning.Each language is evaluated separately to assess the benefit of multilingual fine-tuning.
5 Related Work
Related work spans multimodal vision-language pre-training and multilingual Transformer pre-training, while LayoutXLM builds on both directions for visually-rich documents.
- Multimodal pre-training: ViLBERT, VL-BERT, VisualBERT, and UNITER learn joint or aligned image-text representations for vision-language tasks.These approaches extend Transformer or BERT-style architectures with visual and linguistic inputs.
- Multilingual pre-training: Multilingual models such as mBERT, XLM, XLM-RoBERTa, mBART, mT5, and InfoXLM pre-train Transformers across languages for cross-lingual transfer.The cited models have produced strong results on benchmarks including XNLI and XTREME.
- LayoutXLM: LayoutXLM uses multilingual textual models for initialization and applies them to visually-rich document understanding across languages.This connects multilingual language pre-training with document layout and visual information.
6 Conclusion
The paper presents LayoutXLM and XFUND as resources for multilingual visually-rich document understanding, with experiments reporting strong performance against SOTA baselines.
- Conclusion: LayoutXLM is pre-trained on 30 million scanned and digital-born documents in 53 languages.The model targets multilingual visually-rich document understanding.
- Conclusion: XFUND provides key-value labeled forms in seven languages for multilingual form understanding evaluation.The dataset is introduced alongside LayoutXLM as a benchmark resource.
- Conclusion: Experimental results show LayoutXLM significantly outperforms SOTA baselines for multilingual document understanding.The authors state that the model and XFUND are publicly available to advance research.
- Conclusion: Future work will expand multilingual training data to more languages, document layouts, and templates, and investigate contrastive learning on parallel documents.The proposed direction addresses business documents with the same content in different languages.