PaperlayerView exact revision on arXiv

Source-linked AI summary

LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding

Jiapeng Wang, Lianwen Jin, Kai Ding

arXiv:2202.13669v1cs.CL

TL;DR

Existing structured document understanding models largely focus on single-language data, typically English, limiting multilingual applicability. LiLT separates layout from text during pre-training and recombines them for fine-tuning with off-the-shelf textual models. Across eight languages and three evaluation settings, it achieves competitive or superior performance on diverse downstream benchmarks.

  • Problem

    Most structured document understanding models pre-train and fine-tune on a single language, typically English, despite multilingual application needs.

  • Method

    LiLT decouples text and layout during pre-training, enhances their interaction with BiACM, and re-couples layout with off-the-shelf monolingual or multilingual textual models for fine-tuning.

  • Results

    LiLT achieves competitive or superior performance on diverse downstream benchmarks across eight languages under language-specific, zero-shot transfer, and multitask fine-tuning settings.

  • Takeaways & Limitations

    LiLT provides language-independent benefit from layout pre-training by learning layout knowledge from monolingual structured documents and generalizing it to multilingual ones.

Abstract

from arXiv · show

Structured document understanding has attracted considerable attention and made significant progress recently, owing to its crucial role in intelligent document processing. However, most existing related models can only deal with the document data of specific language(s) (typically English) included in the pre-training collection, which is extremely limited. To address this issue, we propose a simple yet effective Language-independent Layout Transformer (LiLT) for structured document understanding. LiLT can be pre-trained on the structured documents of a single language and then directly fine-tuned on other languages with the corresponding off-the-shelf monolingual/multilingual pre-trained textual models. Experimental results on eight languages have shown that LiLT can achieve competitive or even superior performance on diverse widely-used downstream benchmarks, which enables language-independent benefit from the pre-training of document layout structure. Code and model are publicly available at https://github.com/jpWang/LiLT.

1 Introduction

Structured document understanding is important for intelligent document processing, but existing models largely rely on single-language pre-training. LiLT addresses this limitation by separating reusable layout knowledge from language-specific text and achieves competitive or superior results across eight languages.

  • Structured document understanding supports scanned and digital-born document analysis in intelligent document processing and industries including finance, medicine, and insurance.
  • Existing structured-document models generally pre-train and fine-tune on one language, typically English, limiting use for other languages with little structured-document data.
  • Unchanged layout structure can accommodate language substitution without obvious unnaturalness, motivating the decoupling and reuse of layout invariance across languages.
  • LiLT decouples text and layout during pre-training, re-couples them for fine-tuning, and combines its layout flow with off-the-shelf monolingual or multilingual textual models.
  • LiLT achieves competitive or superior performance on diverse downstream benchmarks across eight languages under varied settings.
  • BiACM enables language-independent cross-modality interaction, while KPL and CAI provide additional pre-training objectives alongside MVLM.

2 LiLT

The method processes document text and layout separately, enables their interaction through BiACM, and combines the resulting features for self-supervised pre-training.

  • An input document image is processed with off-the-shelf OCR engines to obtain text bounding boxes and contents.
  • Text and layout information are separately embedded and passed through corresponding Transformer-based architectures to produce enhanced features.
  • BiACM performs cross-modality interaction, after which encoded text and layout features are concatenated for self-supervised pre-training.

2.1 Model Architecture

LiLT uses parallel text and layout Transformer flows, embedding each modality separately before cross-modal interaction and feature concatenation. Layout embeddings combine normalized box geometry with 1D positional information, while BiACM shares attention knowledge across flows throughout encoding.

  • 2.1.1 Dual-stream Architecture: LiLT uses parallel dual-stream Transformer flows with a reduced layout hidden and intermediate size for computational efficiency.
  • 2.1.1 Text Embedding: Text tokens are ordered by OCR bounding boxes, padded or truncated to length N, and embedded by summing token and 1D positional embeddings before layer normalization.
  • 2.1.2 Layout Embedding: Layout embeddings encode normalized box coordinates, height, and width through four embedding layers, then combine the resulting 2D embedding with 1D positional information.
  • 2.1 Model Architecture: At the model output, text and layout features are concatenated for self-supervised pre-training or downstream fine-tuning.
  • 2.1.3 BiACM: BiACM strengthens cross-modality interaction across the entire encoding pipeline rather than combining text and layout only at the encoder output.
  • 2.1.3 BiACM: BiACM shares same-head, same-layer attention scores between text and layout flows, using detached textual attention during pre-training to preserve text-stream consistency.

2.2 Pre-training Tasks

LiLT uses three self-supervised tasks to learn joint text-layout representations: masked visual language modeling, key point location prediction, and cross-modal alignment identification.

  • Overview: Three self-supervised pre-training tasks guide LiLT to learn joint representations through cross-modal cooperation.
  • MVLM: MVLM masks input tokens and trains the model to recover them over the vocabulary while layout information remains unchanged.
  • MVLM: MVLM masks 15% of text tokens, with 80% replaced by [MASK], 10% by random vocabulary tokens, and 10% unchanged.
  • KPL: KPL divides the layout into 7×7=49 regions, masks bounding boxes, and predicts the regions containing each box’s top-left corner, bottom-right corner, and center.
  • KPL: KPL predicts discretized regions instead of exact locations to moderately relax penalties caused by OCR detection errors while improving performance.
  • Cross-modal Alignment Identification: CAI classifies masked-and-replaced or unchanged token-box pairs as mis-aligned or aligned using an additional head and cross-entropy loss.

2.3 Optimization Strategy

LiLT uses asymmetric optimization during pre-training to preserve the layout flow’s ability to cooperate with different textual models. During fine-tuning, it returns to unified end-to-end optimization and removes BiACM detachment.

  • Pre-training Optimization: A unified learning rate during pre-training would make the layout flow follow the evolving text flow, harming cooperation with different off-the-shelf textual models.
  • Pre-training Optimization: LiLT explores ratios that greatly slow text-stream pre-training updates and finds an appropriate reduction ratio better than parameter freezing.
  • Fine-tuning Optimization: Fine-tuning uses a unified learning rate for end-to-end optimization and cancels BiACM’s DETACH operation.

3 Experiments

Experiments evaluate LiLT’s pre-training choices, downstream performance, and cross-lingual transfer across monolingual and multilingual benchmarks. Results show that BiACM, the proposed pre-training objectives, and layout pre-training support strong performance with monolingual data.

  • 3.2 Ablation Study: BiACM supports cross-modal cooperation while preserving text-flow consistency, whereas co-attention causes severe performance drops in the ablation.Removing DETACH has different effects in pre-training and fine-tuning: it is necessary during pre-training but improves performance during fine-tuning.
  • 3.2 Ablation Study: Both KPL and CAI substantially improve performance, CAI benefits the model more than KPL, and combining them outperforms either task alone.
  • 3.2 Ablation Study: F1 scores rise with the text-flow slow-down ratio until 1000, then fall above 1000; LiLT therefore uses a default ratio of 1000.
  • 3.3 Comparisons with the SOTAs: On monolingual benchmarks, LiLT works with monolingual or multilingual text models and generally outperforms LayoutXLM, while remaining competitive with language-specific models.The reported monolingual datasets are FUNSD, CORD, EPHOIE, and RVL-CDIP.
  • 3.3 Comparisons with the SOTAs: On FUNSD and XFUND, LiLT achieves the highest F1 scores for both SER and RE in each language using 11M monolingual data, and it significantly outperforms competitors in zero-shot transfer.In zero-shot transfer, LiLT has not seen non-English documents, unlike LayoutXLM.

4 Related Work

Earlier structured document understanding methods largely targeted a single language, while LayoutXLM was the only pre-existing multilingual model and required extensive multilingual data processing. LiLT instead transfers layout knowledge from monolingual pre-training to multilingual document understanding.

  • Cross-lingual zero-shot and multitask evaluations compare multilingual transfer under distinct fine-tuning and testing settings.
  • Earlier SDU approaches mainly focused on English, limiting their applicability to multilingual document scenarios.
  • LayoutXLM addressed multilingual SDU with multilingual textual pre-training but required heavy data collection, cleaning, filtering, and pre-training.
  • LiLT pre-trains on monolingual structured documents and transfers layout knowledge to multilingual documents by separating and later recombining layout and text information.

5 Conclusion

LiLT learns layout knowledge from monolingual structured documents and generalizes it to multilingual structured document understanding. The authors report effectiveness across eight languages and three evaluation settings, while identifying broader multilingual visual information as future work.

  • LiLT learns layout knowledge from monolingual structured documents and generalizes it to multilingual structured document understanding.
  • Experimental results across eight languages and language-specific, cross-lingual zero-shot, and multitask fine-tuning settings illustrate LiLT’s effectiveness.
  • Future research will investigate generalized rather than language-specific visual information in multilingual structured documents.

A Dataset Details

The evaluation uses English, Chinese, and multilingual document datasets spanning form understanding, receipt information extraction, examination-paper entity recognition, classification, and cross-lingual form understanding. These datasets provide task-specific annotations, splits, and metrics for comparison.

  • FUNSD contains 199 annotated English scanned forms for semantic entity recognition, split into 149 training and 50 test samples.
  • CORD provides 800 training, 100 validation, and 100 test English receipts, with 30 fields across four categories evaluated using entity-level F1.
  • EPHOIE contains 1,494 Chinese examination-paper images, divided into 1,183 training and 311 testing samples across ten entity categories.
  • RVL-CDIP contains 400,000 grayscale English document images across 16 classes, evaluated using overall classification accuracy.
  • XFUND contains 1,393 annotated forms in seven languages, supporting semantic entity recognition and key-value relation extraction with F1 evaluation.

B Fine-tuning Details

Fine-tuning adds task-specific prediction heads to LiLT representations for semantic entity recognition, document classification, and relation extraction. The procedures use token labels, explicit visual features, or entity-pair representations according to each task.

  • Fine-tuning for Semantic Entity Recognition: Semantic entity recognition uses a token-level classification layer to predict BIO tags for entity fields on four datasets.
  • Fine-tuning for Document Classification: Document classification concatenates pooled ResNeXt101-FPN visual features with the [CLS] representation before final classification.
  • Fine-tuning for Relation Extraction: Relation extraction constructs candidate pairs of semantic entities and represents each pair using head/tail token vectors and entity-type embeddings.
Loading 2202.13669v1…