Source-linked AI summary
StrucTexT: Structured Text Understanding with Multi-Modal Transformers
Yulin Li, Yuxi Qian, Yuchen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, Errui Ding
TL;DR
Structured text understanding in VRDs requires entity labeling and linking across token and segment levels, but existing methods provide limited unified support and visual modeling. StrucTexT combines aligned multi-modal representations with three self-supervised pre-training tasks, and reports superior performance on three real-world benchmarks. The paper also evaluates modality and granularity effects, while noting instability under one sampling strategy.
Problem
Existing structured text understanding methods have limited support for extracting structured data across token and segment levels and often use visual features less elaborately.
Method
StrucTexT uses a unified transformer framework with segment-token alignment, 2D layout embeddings, and MVLM, SLP, and PBD pre-training tasks.
Results
StrucTexT reports superior performance over state-of-the-art methods on three real-world benchmark datasets, with ablations supporting its pre-training strategies.
Takeaways & Limitations
The framework supports structured text extraction at both segment and token granularities while combining text, image, and layout information.
Takeaways & Limitations
Training is unstable when using the described sampling strategy alone, motivating additional handling for imbalanced data.
Abstract
from arXiv · showhide
Structured text understanding on Visually Rich Documents (VRDs) is a crucial part of Document Intelligence. Due to the complexity of content and layout in VRDs, structured text understanding has been a challenging task. Most existing studies decoupled this problem into two sub-tasks: entity labeling and entity linking, which require an entire understanding of the context of documents at both token and segment levels. However, little work has been concerned with the solutions that efficiently extract the structured data from different levels. This paper proposes a unified framework named StrucTexT, which is flexible and effective for handling both sub-tasks. Specifically, based on the transformer, we introduce a segment-token aligned encoder to deal with the entity labeling and entity linking tasks at different levels of granularity. Moreover, we design a novel pre-training strategy with three self-supervised tasks to learn a richer representation. StrucTexT uses the existing Masked Visual Language Modeling task and the new Sentence Length Prediction and Paired Boxes Direction tasks to incorporate the multi-modal information across text, image, and layout. We evaluate our method for structured text understanding at segment-level and token-level and show it outperforms the state-of-the-art counterparts with significantly superior performance on the FUNSD, SROIE, and EPHOIE datasets.
1 INTRODUCTION
Structured text understanding in VRDs must capture entities and relationships across both token and segment levels while using text, image, and layout information. StrucTexT addresses this with a unified multi-modal framework and self-supervised pre-training, achieving superior benchmark performance.
- Motivation: VRD understanding extracts semantic entities and relationships from forms, receipts, and invoices for document intelligence applications.The task supports office automation, accounting, and electronic archiving.
- Motivation: Token-level methods handle character or word labels, whereas segment-level methods capture richer geometry but cannot represent entities composed of partial characters.VRD entities may appear as partial characters, individual segments, or multiple segment lines.
- Motivation: Existing approaches remain limited by incomplete use of visual information and by focusing on only one representation granularity.Prior methods variously use plain text, 2D layout, or shallow multi-modal fusion, while pre-trained models emphasize textual features over elaborate visual features.
- Method: StrucTexT unifies token-level and segment-level extraction through a transformer encoder with segment IDs, 2D position embeddings, and cross-level feature fusion.The framework learns from segment images and word tokens, then combines encoded features across levels and modalities.
- Method: Three self-supervised tasks enrich multi-modal representations: MVLM models textual context, SLP predicts segment length, and PBD identifies relative segment direction.Together, the tasks use textual, visual, and layout features during pre-training.
- Results: Extensive experiments report superior StrucTexT performance over state-of-the-art methods, while ablations support the effectiveness of its pre-training strategies.The paper evaluates the framework on real-world structured text understanding benchmarks.
2 RELATED WORK
Prior structured text extraction methods commonly rely on single-granularity or primarily textual representations. The paper motivates a unified approach that supports both token- and segment-level extraction with multi-modal features.
- Structured Text Understanding: Structured text understanding comprises entity labeling for named entities and entity linking for semantic key-value relationships.These tasks retrieve structured data from visually rich documents.
- Structured Text Understanding: Many existing approaches formulate entity labeling as token-level sequential labeling after serializing OCR text into a linear sequence.NER models commonly assign IOB tags to words or characters.
- Multi-Modal Feature Representations: Other methods represent documents with segment, block, grid, or graph structures and integrate layout and visual information for extraction.Examples include bounding-box sequence reconstruction, visual logical blocks, 2D grids, and multi-modal fusion.
- Multi-Modal Feature Representations: The related-work review identifies a coverage gap: existing methods generally handle only one granularity, motivating unified token-level and segment-level extraction.The proposed framework is intended to support flexible granularity modeling for structured text understanding.
3 APPROACH
StrucTexT embeds text, visual segments, and layout, then uses transformer-based cross-modal fusion and self-supervised objectives to support structured extraction across granularities.
- Multi-Modal Feature Embedding: StrucTexT combines text, image, layout, and segment-ID embeddings before transformer encoding.Text tokens are paired with layout features, while visual segment features come from CNN maps and RoIAlign; segment IDs align modalities across granularity levels.
- Multi-Modal Feature Enhance Module: Three self-supervised tasks jointly learn multimodal representations: MVLM captures contextual language information, SLP predicts segment length, and PBD predicts spatial direction.SLP transfers language-sequence information through shared segment IDs, while PBD divides 360-degree relative positions into eight directional buckets.
- Multi-Modal Feature Enhance Module: PBD uses differences between transformer-encoded visual features to classify relative positional directions between segment pairs.The task computes pairwise segment angles and uses the visual-feature difference as its classification input.
- Cross-granularity Labeling Module: The model supports both token-level and segment-level entity labeling by aggregating tokens sharing a segment ID.Segment text features are computed by averaging token features, fused with visual features through a Hadamard product, and classified with cross-entropy loss.
- Segment Relationship Extraction Module: The segment relationship module predicts directed links between segments for entity linking and balances positive and negative pairs through sampling.An asymmetric parameter matrix produces link probabilities, while equal positive and negative sampling addresses the prevalence of unrelated segment pairs.
- Segment Relationship Extraction Module: Training becomes unstable with the sampling strategy alone, so the linking objective combines Margin Ranking Loss and Binary Cross-Entropy.The combined losses are introduced to handle the imbalanced distribution of entity linking.
4 EXPERIMENTS
StrucTexT is evaluated across segment-level and token-level entity tasks on FUNSD, SROIE, and EPHOIE, with additional ablations of its pre-training components. It achieves strong benchmark performance and shows that the proposed self-supervised tasks improve results.
- Datasets and setup: The experiments cover segment-level entity labeling and linking, token-level entity labeling, and ablations on FUNSD, SROIE, and EPHOIE.The model is evaluated on three downstream benchmarks, while DOCBANK and RVL-CDIP are used for pre-training.
- Segment-level entity labeling: 95.84% precision and 98.52% recall on SROIE yielded a 0.27% F1-score gain over LayoutLMv2_LARGE for segment-level entity labeling.The reported comparison is against the LayoutLMv2_LARGE baseline.
- Segment-level entity labeling: 83.09% F1-score was achieved on FUNSD segment-level entity labeling, while LayoutLMv2_BASE was outperformed under the same architecture settings.LayoutLMv2_LARGE exceeded StrucTexT by approximately 1% but used a larger model and substantially more pre-training documents.
- Segment-level entity linking: StrucTexT achieved 44.1% F1-score for FUNSD entity linking, with 78.36% mAP, 79.19% Hit@1, 84.33% Hit@2, and 95.33% Hit@5.The method improved over SPADE by 2.4% and reported a competitive mRank score of 3.38.
- Token-level entity labeling: 97.95% was reported for token-level entity labeling on EPHOIE, whose annotations are character-based.The evaluation applies token-level prediction to entities annotated at the character level.
- Ablation study: MVLM+PBD improved performance by nearly 3% on FUNSD and 0.2% on SROIE compared with MVLM alone, while all three tasks achieved the best combination.The ablation attributes richer cross-modal semantics and potential relationships to the SLP and PBD tasks.
5 CONCLUSION
The paper concludes that StrucTexT provides a unified framework for structured text understanding by combining multimodal and multi-granularity feature fusion. Its experiments report superior performance on three real-world benchmarks and examine how modalities and granularities affect entity extraction.
- Conclusion: StrucTexT uses a unified framework to improve document text-structure understanding through multimodal and multi-granularity feature fusion.The conclusion presents this framework as the paper’s central approach.
- Conclusion: The framework shows superior performance on three real-world benchmark datasets after applying novel pre-training strategies.The conclusion links the reported benchmark performance with the pre-training strategies and feature fusion.
- Conclusion: The study evaluates how different modalities and granularities influence entity extraction and uses this analysis to provide a perspective on structured text understanding.This conclusion is limited to the paper’s evaluation of modality and granularity effects.