Source-linked AI summary
StructuralLM: Structural Pre-training for Form Understanding
Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, Luo Si
TL;DR
Text-only pretrained models overlook cell-level layout information needed for scanned document understanding. StructuralLM jointly models cells and layout through cell-level representations and cell position classification, achieving new state-of-the-art results across three downstream tasks.
Problem
Text-only pretrained models neglect cell-level layout information that is important for understanding scanned forms and other structured documents.
Method
StructuralLM jointly models cells and layout using shared cell-level 2D-position embeddings and a cell position classification pre-training objective.
Results
StructuralLM achieves new state-of-the-art results across form understanding, document visual question answering, and document image classification.
Takeaways & Limitations
Jointly representing cell semantics and layout interactions supports downstream scanned-document understanding tasks.
Abstract
from arXiv · showhide
Large pre-trained language models achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, they almost exclusively focus on text-only representation, while neglecting cell-level layout information that is important for form image understanding. In this paper, we propose a new pre-training approach, StructuralLM, to jointly leverage cell and layout information from scanned documents. Specifically, we pre-train StructuralLM with two new designs to make the most of the interactions of cell and layout information: 1) each cell as a semantic unit; 2) classification of cell positions. The pre-trained StructuralLM achieves new state-of-the-art results in different types of downstream tasks, including form understanding (from 78.95 to 85.14), document visual question answering (from 72.59 to 83.94) and document image classification (from 94.43 to 96.08).
1 Introduction
StructuralLM addresses the need to model cell-level semantics and spatial relationships in scanned forms and tables, achieving stronger downstream results than existing models.
- Scanned documents organize textual information in layouts including multicolumn formats, tables, forms, receipts, and invoices.
- Word-level layout modeling can split words from one semantic cell into separate entities, as when LayoutLM separates “LORILLARD” and “ENTITIES.”The paper states that the combined “LORILLARD ENTITIES” should be predicted as an Answer entity.
- Words from the same cell often express meaning together, so document models must capture cell semantics and spatial relationships between cells.
- StructuralLM jointly exploits cell and layout information by sharing cell-level 2D-position embeddings across tokens in the same cell.
- StructuralLM outperforms state-of-the-art models on form understanding, document visual question answering, and document image classification.Reported scores improve from 78.95 to 85.14, 72.59 to 83.94, and 94.43 to 96.08, respectively.
2 StructuralLM
StructuralLM extends a BERT-based encoder with cell-level layout representations and two self-supervised objectives for scanned document understanding.
- 2.1 Model Architecture: StructuralLM uses BERT as its attention-based bidirectional Transformer backbone for adapting pretrained language modeling to document images.
- 2.2 Cell-level Input Embedding: StructuralLM combines word embeddings, cell-level 2D-position embeddings, and 1D-position embeddings before contextual processing by a bidirectional Transformer.Tokens from the same cell share 2D positions, while 1D positions preserve token order within the cell.
- 2.2 Cell-level Input Embedding: The model uses cell-level 2D positions so it can identify words from the same cell and derive contextualized cell representations.
- 2.3 Pre-training Tasks: Masked visual-language modeling masks tokens while retaining their cell positions, training the encoder to predict the masked tokens using text and layout context.
- 2.3 Pre-training Tasks: Cell position classification divides an image into N equal areas and predicts the area containing a selected cell using an encoder classification layer.Cells with masked tokens are excluded from this task when the two pre-training objectives are performed simultaneously.
- 3 Experiments: The pretrained model is fine-tuned for form understanding, document visual question answering, and document image classification.Form understanding uses sequential token labeling, while image classification uses the [CLS] representation.
3 Experiments
StructuralLM is evaluated through pre-training configuration, fine-tuning on three document-image tasks, and ablations of its cell-level design. It outperforms prior models across form understanding, document visual question answering, and document classification, while both proposed components contribute to performance.
- Fine-tuning on Form Understanding: 6% F1 point improvement over LayoutLM makes StructuralLM the best model on the FUNSD test set under the same model size.The comparison covers BERT, RoBERTa, LayoutLM, and StructuralLM using Precision, Recall, and F1 evaluation.
- Fine-tuning on Document Visual QA: Over 11% ANLS improvement over LayoutLM is achieved on DocVQA, rising to over 14% on the Form&Table subset.ANLS is the evaluation metric for the DocVQA test set and its Form&Table subset.
- Fine-tuning on Document Classification: StructuralLM achieves higher RVL-CDIP classification accuracy than image-based and text-based approaches, as well as LayoutLM.The task uses overall classification accuracy on 16 document classes.
- Ablation Study: Removing cell-level layout embeddings lowers FUNSD F1 from 0.8514 to 0.8024, while removing cell position classification lowers it to 0.8125.These ablations evaluate the individual contributions of the two StructuralLM components.
- Ablation Study: Cell-level 2D-position embeddings improve F1 from 0.7895 to 0.8125 over LayoutLM and yield lower masked visual-language-model training loss.The comparison is made after ablating cell position classification and under the same pre-training settings.
4 Related Work
Document understanding research has progressed from statistical parsing to deep learning and self-supervised pre-training that jointly models document text and layout.
- 4.1 Statistical Approaches: Earlier document segmentation methods treated layout analysis as parsing and used grammar-based losses with machine learning for feature selection and parameter training.
- 4.2 Deep Learning Approaches: Deep learning approaches address document structure extraction using multimodal networks, segmentation and bounding-box prediction, and contextual region detection.These methods include pixel-level classification, fully convolutional encoder-decoder networks, and Faster R-CNN with contextual information.
- 4.3 Pre-training Approaches: Self-supervised pre-training methods learn interactions between text and layout for scanned document understanding tasks.LayoutLM incorporates visual information during fine-tuning and achieves state-of-the-art results on several downstream tasks.
5 Conclusion
StructuralLM is a structural pre-training approach that jointly models cell and layout information in scanned documents. It uses cell-level positional embeddings and cell position classification, and achieves new state-of-the-art results across downstream tasks.
- StructuralLM outperforms strong baselines and achieves new state-of-the-art results across downstream document understanding tasks.
- StructuralLM represents tokens from the same cell with shared cell-level 2D positions and adds cell position classification to learn interactions between cells and layout.The approach is built on an extension of the Transformer encoder and is pre-trained on large unlabeled documents.