Source-linked AI summary
BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents
Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, Sungrae Park
TL;DR
KIE requires models to understand text and layout despite imperfect text ordering and expensive annotation. BROS combines relative 2D spatial encoding with area-masked pre-training without visual features, achieving comparable or better performance while remaining robust in ordering-imprecise and data-scarce settings.
Problem
Document KIE depends on spatial semantics, but complex layouts lack reliable text ordering and labeled examples are expensive.
Method
BROS combines relative spatial encoding between text blocks with area-masked language modeling for 2D documents.
Results
BROS achieves comparable or better performance without visual features and shows robust performance under imprecise serialization and limited training data.
Takeaways & Limitations
Text-and-layout modeling with 2D pre-training can match or exceed approaches that incorporate additional visual features.
Takeaways & Limitations
Sequence-based BIO tagging requires correct text-block order and cannot represent one-to-many relationships between blocks.
Abstract
from arXiv · showhide
Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods. Code is available at https://github.com/clovaai/bros.
Introduction
Document KIE must combine textual and spatial information, but serialized text can lose layout information and depend on imperfect reading order. BROS addresses these issues by encoding relative text-block positions and using area-masked pre-training, while targeting robustness to ordering errors and data scarcity.
- Motivation: Document KIE extracts ordered items, prices, taxes, and key-value pairs from varied industrial-document layouts.The task combines computer-vision and natural-language-processing components because documents have diverse layouts.
- Challenges: Complex layouts such as multi-column documents and multiple tables make proper text-block ordering non-trivial, with no general-purpose serializer available.This motivates parsing modules that can handle incomplete or incorrect reading orders.
- Motivation: Converting 2D text blocks into 1D sequences loses layout information that is critical for KIE.Early systems therefore depend heavily on accurate document parsing and text-block order.
- Approach: BROS encodes relative positions between text blocks and introduces area-masked language modeling without relying on visual features.Its area-masking strategy hides texts within a document area to reflect the two-dimensional structure of text blocks.
- Evaluation focus: BROS addresses real-world KIE challenges involving incorrect text ordering and limited labeled data.The paper evaluates robustness to serialization errors and fine-tuning efficiency in data-scarce settings.
Related Work
BROS builds on layout-aware document language models while replacing absolute spatial encoding with relative positions and adding 2D area-based pre-training. Its design also uses graph-based parsing to address ordering and token-relationship limitations in document KIE.
- Layout-aware pre-training: LayoutLM jointly models text and layout by encoding absolute text-block positions, while later models additionally integrate visual features.LayoutLM is described as a strong baseline for multiple KIE tasks.
- Parsers for document KIE: SPADE replaces sequence-based BIO parsing for settings where text-block order is unreliable or token relationships must be represented explicitly.It creates a directed relation graph linking tokens to represent entities and their relationships.
- Relative spatial encoding: BROS encodes relative positions between text blocks directly in Transformer attention instead of relying on absolute positions.The relative representation is based on differences between corresponding bounding-box vertices and is transformed before attention use.
- Relative spatial encoding: Relative positions provide co-modality for semantically coupled text blocks regardless of their absolute locations.The comparison uses paired fields such as “Project #” and “Total Sample” with their values.
- Relative spatial encoding: BROS combines relative spatial embeddings with token semantics and incorporates the relative shape and size of text blocks through all four vertices.Its embeddings are shared across attention heads, while the semantic and spatial features are combined in the attention logit.
- 2D masking objectives: BROS pre-trains with token-masked language modeling and area-masked language modeling, which masks all text blocks within a randomly expanded 2D area.The area-masking procedure selects a block, expands its region, identifies included blocks, and predicts their masked tokens.
Key Information Extraction Tasks
KIE covers entity extraction and entity linking in document layouts, evaluated on four benchmarks that model forms, receipts, and related documents. Because standard datasets provide text-block order, the paper also removes that information to reflect less orderly real scenarios.
- Entity extraction (EE) identifies text-block sequences representing target texts, while entity linking (EL) connects key entities through hierarchical or semantic relations.
- Figure 5 illustrates EE with colored entity blocks and EL with red arrows showing hierarchical relationships.
- The benchmarks include FUNSD, SROIE*, CORD, and SciTSR, covering forms, receipts, and other document KIE settings.
- Because the benchmarks generally provide text-block order, the authors remove that order to represent scenarios where serialization is imperfect.
Experiments
Experiments evaluate BROS under standard ordering, missing or permuted text-block order, varying fine-tuning data, and component ablations. Across these settings, BROS generally outperforms or matches competing layout-aware models while using text and layout without visual features.
- With the Order Information of Text Blocks: On FUNSD EE, BROS improves over the previous best by 2.51 F1 points for BASE models and 5.57 for LARGE models.The reported changes are 80.54 →83.05 and 78.95 →84.52, respectively.
- With the Order Information of Text Blocks: BROS achieves the best performance on all reported ordered EE and EL tasks except SciTSR, while BERT performs worst.LayoutLM and LayoutLMv2 outperform BERT by encoding layout, and LayoutLMv2 often benefits from visual features.
- Without the Order Information of Text Blocks: BROS performs best on orderless benchmarks except p-SciTSR and suffers only minor drops, while LayoutLM* and LayoutLMv2* show large degradations.
- Without the Order Information of Text Blocks: BROS remains relatively consistent across multiple FUNSD serializers, whereas LayoutLM* and LayoutLMv2* degrade as text-block order becomes less reasonable.LayoutLM* LARGE combined with SPADE shows unstable performance of 48.30±12.51.
- Fine-tuning Data Efficiency: BROS performs best on FUNSD tasks across fine-tuning ratios from 10% to 100% and when trained with only 5 or 10 examples.All models tend to improve as the training-data ratio increases; the authors interpret the few-example results as evidence of generalization.
- Ablation Study: Adding relative positional encoding improves LayoutLM by 3.62pp on average, adding the revised pre-training objectives by 1.14pp, and combining both by 5.10pp.
- Ablation Study: Relative positional methods outperform absolute position encoding, with larger gaps in entity linking, and BROS’s method performs best among them.
Conclusion
BROS models document text and layout in 2D space using relative positions and area-masked pre-training. It delivers robust key information extraction without additional visual features, including under imprecise serialization and limited training data.
- BROS encodes texts in 2D space with relative positions and uses area-masking pre-training without additional visual features.
- BROS shows robust performance under imprecise text serialization and small amounts of downstream training data, while other models degrade significantly.
Two Categories of Document KIE tasks and Parsers for Them
Document KIE comprises entity extraction and entity linking, addressed by sequence-based BIO tagging and order-independent graph-based SPADE decoding. SPADE identifies and connects tokens for extraction and directly models token relations for linking.
- Two Categories of Document KIE tasks: Document KIE includes entity extraction, which identifies target text-block sequences, and entity linking, which connects key entities through hierarchical or semantic relations.
- BIO Tagger: BIO tagging identifies beginning and inside labels in an ordered text-block sequence, so incorrect ordering can prevent successful key-information extraction.
- BIO Tagger: The BIO tagger cannot represent links between text blocks as sequence units and does not handle one-to-many relations well.
- SPADE Decoder: SPADE extracts a directional sub-graph from a fully connected graph, enabling extraction without text-block order information.
- SPADE Decoder: For entity extraction, SPADE classifies initial tokens and connects subsequent tokens using token representations from the pre-trained model.
- SPADE Decoder: For entity linking, SPADE performs binary classification over token pairs, allowing one token to hold multiple relations with other tokens.
KIE Benchmark Datasets
The paper evaluates document KIE methods across four benchmark datasets covering entity extraction and linking. The supplied materials also identify comparison tables for model performance and inference speed.
- Datasets: The benchmark suite contains FUNSD, SROIE*, CORD, and SciTSR, covering three entity-extraction and three entity-linking tasks.
- FUNSD: FUNSD contains 149 training and 50 testing examples, with Header, Question, and Answer entity classes and header-question and question-answer relations.
- Dataset Samples: Figure 9 presents sample images from the four benchmark datasets.
- Evaluation Materials: The evaluation materials include tables comparing LayoutLM implementations, BIO tagger and SPADE decoder performance, and inference speed.
Compare Published LayoutLM Model and Our Own Implementation
The authors compare their LayoutLM† implementation with published LayoutLM scores on the FUNSD entity-extraction task across different pre-training-data settings. LayoutLM† achieves comparable performance across all supplied settings.
- LayoutLM† shows comparable performance to reported LayoutLM scores across all tested pre-training-data settings on FUNSD entity extraction.
Compare BIO Tagger and SPADE Decoder
With text-block order given, BIO tagging and SPADE decoding show little overall performance difference across the three EE tasks, while BROS remains strongest across parsers.
- BIO tagger and SPADE decoder perform similarly across the three EE tasks when text-block order is provided.
- BERT performs better on FUNSD and CORD with SPADE decoding than with BIO tagging.The passage attributes this difference to SPADE’s use of 2D layout information.
- BROS achieves the best performance in all tasks regardless of the parser.
Compare the Inference Speed of the Models
BROS trades speed for spatial modeling: it is slower than LayoutLM but faster than image-feature-based LayoutLMv2, while achieving the best performance among the compared models.
- BROS is slower than LayoutLM because it considers relative positions for all text-block pairs.
- BROS is faster than LayoutLMv2, which uses image features.
- BROS shows the best performance among the compared models.
Experiments on Rotated Images
The rotated-image experiment tests BROS under imprecise text serialization by rotating documents and re-serializing their text blocks, with results following the same trend as Table 6.
- The experiment uses rotated images in which the image and locations of all text blocks are rotated.
- Text-block order is re-serialized using the yx- setup to create another imprecise serialization condition.
- Figure 10 reports the same trend as Table 6.