PaperlayerView exact revision on arXiv

Source-linked AI summary

ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding

Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Teng Hu, Weichong Yin, Yongfeng Chen, Yin Zhang, Shikun Feng, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

arXiv:2210.06155v2cs.CLcs.AI

TL;DR

Document pre-training often lacks systematic use of layout knowledge, including reading order and layout’s role alongside text and images. ERNIE-Layout combines order-aware serialization and pre-training with spatial-aware attention; the paper reports improved downstream performance and state-of-the-art results on several tasks.

  • Problem

    Existing document pre-training often uses raster-scan ordering and treats layout as a position feature, despite layout reflecting reading order and contributing to semantic expression.

    Source: §1 · p. 1

  • Method

    ERNIE-Layout reorders document tokens, adds reading order prediction and replaced regions prediction, and uses spatial-aware disentangled attention in a multimodal transformer.

    Source: §1 · p. 2

  • Results

    ERNIE-Layout achieves new state-of-the-art on FUNSD, CORD, and Kleister-NDA, with a 7.98% improvement over previous best results on FUNSD.

    Source: §4.3 · p. 7

  • Takeaways & Limitations

    The conclusion reports considerable downstream improvement over raster-scan ordering and says experiments demonstrate the effectiveness of integrating layout knowledge into document pre-training.

    Source: §5 · pp. 8–9

Abstract

from arXiv · show

Recent years have witnessed the rise and success of pre-training techniques in visually-rich document understanding. However, most existing methods lack the systematic mining and utilization of layout-centered knowledge, leading to sub-optimal performances. In this paper, we propose ERNIE-Layout, a novel document pre-training solution with layout knowledge enhancement in the whole workflow, to learn better representations that combine the features from text, layout, and image. Specifically, we first rearrange input sequences in the serialization stage, and then present a correlative pre-training task, reading order prediction, to learn the proper reading order of documents. To improve the layout awareness of the model, we integrate a spatial-aware disentangled attention into the multi-modal transformer and a replaced regions prediction task into the pre-training phase. Experimental results show that ERNIE-Layout achieves superior performance on various downstream tasks, setting new state-of-the-art on key information extraction, document image classification, and document question answering datasets. The code and models are publicly available at http://github.com/PaddlePaddle/PaddleNLP/tree/develop/model_zoo/ernie-layout.

1 Introduction

The introduction argues that document pre-training underuses layout knowledge, especially for reading order and interaction across modalities. ERNIE-Layout addresses these gaps through serialization, pre-training tasks, and spatial-aware attention.

  • Prior document pre-training often treats 2D coordinates as extended 1D positions, while raster-scan ordering can conflict with human reading in complex layouts.The introduction identifies tables, forms, and multi-column templates as examples, and notes that layout may also carry semantic information beyond position.Source: §1 · pp. 1–2
  • ERNIE-Layout significantly outperforms strong baselines on almost all tasks, and the paper reports refreshed state-of-the-art results across various VrDU tasks.Source: §1 · p. 2
  • ERNIE-Layout rearranges input tokens during serialization and introduces reading order prediction, described as the first attempt to model proper reading order in document pre-training.Source: §1 · p. 2
  • The model combines spatial-aware disentangled attention with replaced regions prediction to facilitate fine-grained interaction across text, visual, and layout modalities.Source: §1 · p. 2

2 Related Work

Related work traces layout-aware pre-training from layout embeddings and multimodal tasks to approaches that model spatial relations. It argues that raster-scan serialization and treating layout as a subsidiary text feature leave important opportunities unaddressed.

  • LayoutLM embeds 2D coordinates and extends masked language modeling, while LayoutLMv2 adds image patches and text-image matching and alignment tasks (Xu et al., 2021).Source: §2 · p. 2
  • Other approaches use segment-level layout features (Li et al., 2021a), shared spatial embeddings (Appalaraju et al., 2021), or encoder-decoder generation (Powalski et al., 2021).Source: §2 · p. 2
  • The authors identify raster-scan serialization and treating layout as a subsidiary text feature as limitations; LayoutReader models reading order but is designed for detection rather than directly empowering diverse document tasks (Wang et al., 2021c).Source: §2 · p. 3
  • Knowledge-enhanced language models incorporate external facts or human cognitive patterns, and ERNIE-ViL uses scene-graph knowledge; this paper draws on implicit layout knowledge such as reading order (Sun et al., 2019).Source: §2 · p. 3

3 Methodology

ERNIE-Layout combines layout-aware serialization, text and visual representations, spatial-aware attention, and four pre-training tasks to model document text, layout, and image information.

3.1 Serialization Module

  • Document-Parser serialization yields lower GPT-2 perplexity than raster-scanning order, particularly for documents with complex layouts.Perplexity is the evaluation metric used to compare the two input sequences.Source: Figure 2 · p. 4, §3.1 · p. 4
  • Document-Parser detects document elements and uses element-specific algorithms to infer logical relationships between OCR words and their proper reading order.This addresses cases where raster scanning scatters words from the same table cell.Source: §3.1 · p. 4

3.2 Input Representation

  • Text and visual tokens combine modality embeddings with layout embeddings; Faster R-CNN extracts image features from 224×224 inputs pooled into a 7×7 feature map.The resulting textual and visual embeddings are integrated into a joint sequence.Source: §3.2 · pp. 4–5

3.3 Multi-modal Transformer

  • Spatial-aware disentangled attention includes content and relative 1D, 2D x-axis, and 2D y-axis features in its attention-score calculation.This explicitly incorporates layout features into multimodal attention.Source: §3.3 · p. 5

3.4 Pre-training Tasks

  • ROP models next-token reading-order links, RRP replaces 10% of image patches, and MVLM and TIA add text recovery and text-image coverage prediction.TIA predicts whether each text token is covered by a region masked on the document image.Source: §3.4 · pp. 5–6

4 Experiments

Across key information extraction, document question answering, and document image classification, ERNIE-Layout reports strong downstream performance. The analysis examines its pre-training tasks, attention mechanisms, and serialization modules.

4.1 Datasets

  • The evaluation covers four key information extraction datasets, DocVQA for question answering, and RVL-CDIP for document image classification.Pre-training uses 10 million pages randomly selected from more than 30 million scanned pages.Source: §4.1 · p. 6

4.2 Settings

  • ERNIE-Layout uses a 24-layer transformer initialized from RoBERTa_large and a Faster-RCNN-initialized visual encoder, with downstream-task-specific prediction heads.Pre-training uses 512 textual tokens and 49 visual tokens; fine-tuning uses sequence labeling for key information extraction, extractive QA for DocVQA, and classification for RVL-CDIP.Source: §4.2 · p. 6

4.3 Results

  • ERNIE-Layout sets new state of the art on FUNSD, CORD, and Kleister-NDA, is competitive on SROIE and DocVQA, and refreshes the best RVL-CDIP performance.The FUNSD improvement over previous best results is 7.98%, with standard deviation 0.0011.Source: Table 3 · p. 7, Table 4 · p. 7, Table 5 · p. 7

4.4 Analysis

  • On FUNSD, replaced regions prediction adds 0.95% and reading order prediction further adds 1.3%; spatial-aware self-attention and disentangled attention also improve downstream performance.Layout-knowledge-based serialization improves performance even without disentangled attention.Source: §4.4 · p. 8, Table 6 · p. 8, Table 7 · p. 8

5 Conclusion

ERNIE-Layout integrates layout knowledge through serialization and attention. Experiments and analyses demonstrate its effectiveness and examine different ways of using layout knowledge in visually-rich document understanding.

5 Conclusion

  • ERNIE-Layout integrates layout knowledge into document pre-training through serialization and attention.The model rearranges recognized document words and uses an attention mechanism intended to improve interaction between text/image and layout features.Source: §5 · pp. 8–9

A.1 More Details about Document-Parser

The Document-Parser uses layout parsing and table parsing to reorder recognized words, addressing limitations of raster-scan serialization on complex layouts.

A.1 More Details about Document-Parser

  • Document-Parser combines document-specific OCR, Layout-Parser, and Table-Parser; the latter two are described as crucial for incorporating layout knowledge.Layout and table structures are detected spatially, while paragraph analysis combines heuristics and detection models to identify paragraph layout and upper-lower boundaries.Source: §A.1 · p. 11
  • For the complex-layout example, Document-Parser produces a sequence described as more consistent with human reading habits than raster scanning.The raster-scan sequence repeats “Session Chair,” whereas the parser example includes “Tuula” and “Frank Markert.”Source: Figure 4 · p. 12, §A.1 · p. 12
  • Across 10,000 documents, Document-Parser reduces average serialized-sequence PPL by about 1 point, while showing greater advantages on complex-layout documents.The paper notes that such documents are a small proportion of the total; Table 8 compares Raster-scanning order with Document-Parser.Source: §A.1 · pp. 11–12, Table 8 · p. 11

A.2 More Details about Multi-modal Transformer

The section presents formulas describing spatial-aware disentangled attention and supplements them with a flow chart of the calculation.

A.2 More Details about Multi-modal Transformer

  • A flow chart supplements the formulas describing spatial-aware disentangled attention for the multi-modal transformer.Source: §A.2 · p. 12

A.3.1 Finetuning Datasets

The fine-tuning evaluation covers form, receipt, legal-document, document-classification, and document-question-answering datasets.

FUNSD

  • FUNSD contains 199 annotated scanned forms; evaluation uses official OCR annotations and entity-level F1, with cell-level layout information during fine-tuning.Its training set has 149 samples and its test set has 50.Source: §A.3.1 · p. 12

CORD

  • CORD comprises Indonesian receipts with OCR box/text annotations and semantic labels; its train, validation, and test sets contain 800, 100, and 100 receipts.The evaluation uses official OCR annotations and entity-level F1.Source: §A.3.1 · p. 12

SROIE

  • SROIE contains scanned receipts for extracting company, date, address, and total, with 626 training samples and 347 test samples.The evaluation uses official OCR annotations and entity-level F1.Source: §A.3.1 · p. 12

Kleister-NDA

  • Kleister-NDA covers scanned and born-digital long formal documents and extracts date, jurisdiction, party, and term.Its train, validation, and test sets contain 254, 83, and 203 samples; because the test set is not public, entity-level F1 is reported on validation.Source: §A.3.1 · pp. 12–13

RVL-CDIP

  • RVL-CDIP classifies grayscale document images into 16 classes, using Microsoft OCR for text and layout and classification accuracy for evaluation.Its training, validation, and test sets contain 320,000, 40,000, and 40,000 images.Source: §A.3.1 · p. 13

DocVQA

  • DocVQA evaluates answer-span prediction on 50,000 questions over 12,767 document images, using Average Normalized Levenshtein Similarity.The images are split into training, validation, and test sets at an 8:1:1 ratio; Microsoft OCR supplies text and layouts.Source: §A.3.1 · p. 13

A.3.2 Baselines

The RoBERTa-base ERNIE-Layout model improves on various visually-rich document understanding tasks, while the experiments compare it with text-only and multimodal models.

A.3.2 Baselines

  • The comparisons include BERT, RoBERTa, and UniLMv2 as text-only models, and LayoutLM, LayoutLMv2, TILT, StructuralLM, and DocFormer as multimodal models.ERNIE-Layout and StructuralLM use RoBERTa initialization; the other listed models are initialized from BERT, UniLMv2, or T5.Source: §A.3.2 · p. 13

A.3.3 Results with RoBERTa-base

  • The base-level ERNIE-Layout uses RoBERTa-base and 12 transformer layers with 768 hidden units and 12 attention heads; StructuralLM is omitted because its base-model parameters and results are unavailable.The text notes that StructuralLM does not release its base model parameters and performance.Source: §A.3.3 · p. 13
  • ERNIE-Layoutbase improves performance across various VrDU tasks, especially FUNSD and Kleister-NDA, but is slightly poorer on DocVQA and beats some large models on several datasets.The reported large-model comparisons include FUNSD, CORD, Kleister-NDA, and RVL-CDIP.Source: §A.3.3 · p. 13

Related papers

Loading 2210.06155v2…