Source-linked AI summary
Graph Convolution for Multimodal Information Extraction from Visually Rich Documents
Xiaojing Liu, Feiyu Gao, Qiong Zhang, Huasha Zhao
TL;DR
VRD entity extraction requires visual and layout context that text-only sequence models do not capture. The paper uses graph convolution to combine textual and visual context with text embeddings, and reports that the resulting system outperforms BiLSTM baselines on two real-world datasets.
Problem
VRD semantics depend on visual cues such as layout and relative position, making text-only extraction inadequate for distinguishing entities with similar textual forms.
Method
Graph convolution encodes each text segment’s textual and visual context into graph embeddings, which are combined with text embeddings for BiLSTM-CRF extraction.
Results
The system outperforms BiLSTM baselines on two manually annotated real-world VRD datasets.
Takeaways & Limitations
Graph-based modeling provides a novel approach for information extraction from visually rich documents.
Abstract
from arXiv · showhide
Visually rich documents (VRDs) are ubiquitous in daily business and life. Examples are purchase receipts, insurance policy documents, custom declaration forms and so on. In VRDs, visual and layout information is critical for document understanding, and texts in such documents cannot be serialized into the one-dimensional sequence without losing information. Classic information extraction models such as BiLSTM-CRF typically operate on text sequences and do not incorporate visual features. In this paper, we introduce a graph convolution based model to combine textual and visual information presented in VRDs. Graph embeddings are trained to summarize the context of a text segment in the document, and further combined with text embeddings for entity extraction. Extensive experiments have been conducted to show that our method outperforms BiLSTM-CRF baselines by significant margins, on two real-world datasets. Additionally, ablation studies are also performed to evaluate the effectiveness of each component of our model.
1 Introduction
VRD information extraction must combine text with visual and layout cues because text sequences alone omit document context. The paper proposes graph convolution to summarize textual and visual context before BiLSTM-CRF entity extraction.
- VRDs encode semantic structure through text plus visual features such as layout, tabular structure, and font size.
- The proposed method represents VRDs as graphs of text segments whose positions are defined by bounding-box coordinates.Fonts and colors are identified as potentially useful complementary visual features but remain outside this paper’s scope.
- Graph convolution produces context-aware graph embeddings that are combined with text embeddings for entity extraction using BiLSTM-CRF.
- 1.1 Challenges: Text alone cannot distinguish entities with similar textual forms, such as multiple dates or money amounts in the same document.Correct extraction requires relative positions and visual features to identify roles such as Invoice Date, Transaction Date, Due Date, or tax amount.
- 1.1 Challenges: Template-matching approaches are difficult to scale across thousands of vendor templates, distorted user images, and scenario-specific engineering requirements.
- Experiments on two real-world datasets compare the method with BiLSTM-CRF baselines and use ablations to assess component effectiveness.
2 Related Works
Prior work includes graph neural networks and visual document extraction, but existing approaches often omit visual information or rely on heuristics and known templates. This paper models VRD text segments with graph convolution and explicit edge embeddings.
- Graph neural network research generalizes convolution to graph-structured data, and this paper follows direct graph convolution to model VRD text-segment graphs.
- The proposed graph convolution introduces explicit edge embeddings to model relationships between vertices directly.Self-attention handles variable-sized neighborhoods, and pairwise operations are parallelizable.
- Earlier VRD extraction approaches use visual features but depend on heuristic knowledge, human-designed features, or limited known templates, reducing scalability.
- Graph-based information extraction models capture dependencies or jointly extract entities and relations, but they do not directly model visual information.
3 Model Architecture
The model represents visually rich documents as graphs of text segments, uses graph convolution to encode textual and visual context, and combines these graph embeddings with token embeddings for BiLSTM-CRF entity extraction.
- Entity Extraction: The model concatenates graph embeddings with each token embedding, then feeds the resulting sequence through a BiLSTM, fully connected network, and CRF layer for entity extraction.The graph embedding adds contextual information to the input token sequence.
- Document Representation: Each document is modeled as a graph whose nodes are text segments and whose edges encode visual dependencies such as relative shape and distance.Text segments include their positions and text, and every node is connected to every other node in the experiments.
- Feature Extraction: Node embeddings encode segment text, while edge embeddings represent visual distance, source-node shape, and destination-node relative size.These features capture layout relationships such as the closer positions of related information and differing segment shapes for addresses and names.
- Graph Convolution: Graph convolution operates on node-edge-node triplets rather than nodes alone, producing neighbor features with an MLP and updating node and edge embeddings across layers.The triplet combines visual features with neighbor representations and copies the current node information across neighbors, allowing features to learn where to attend.
- Graph Convolution: Self-attention computes each node’s output representation by attending to its neighbors, with attention coefficients and an activation function controlling aggregation.In the fully connected graph, each node can attend to all other nodes; LeakyReLU is used in the attention mechanism.
4 Model Supervision and Training
The training process uses annotated entity values and locations converted into IOB token labels, while jointly training extraction with an auxiliary segment-classification task.
- Annotation: Documents are annotated with predefined entity values and bounding boxes, then entity text is labeled using the IOB tagging format.Tokens in empty text segments receive the O label.
- Annotation: A manually set overlap threshold assigns annotated entities to OCR text segments when the overlap ratio exceeds that threshold.The ratio uses the overlap area divided by the smaller annotated or OCR box area.
- Training: Graph convolution layers and BiLSTM-CRF extractors are trained jointly, with segment classification added as an auxiliary task to improve prediction accuracy.The classifier predicts a predefined tag from each text segment’s graph embedding.
5 Experiments
Experiments on VATI and IPR compare the graph-convolution model with BiLSTM-CRF baselines using F1 scores. Results show gains especially when entity identification requires visual context, while ablations examine visual features, attention, graph depth, and multi-task learning.
- 5.1 Datasets Description: The experiments use VATI and IPR, with 70% training, 15% validation, and 15% test splits.VATI contains 3,000 Chinese invoices with 16 entities; IPR contains 1,500 English receipts with four entities and 146 templates.
- 5.2 Baselines: The proposed model outperforms both BiLSTM-CRF baselines by significant margins on the two datasets.Baseline I processes each text segment separately, whereas Baseline II concatenates segments into a one-dimensional document sequence.
- 5.3 Results: The model performs almost identically to baselines on text-distinguishable entities but clearly outperforms them on visually dependent entities such as Price, Tax, Buyer, and Seller.Invoice Number and Date are identified from text alone, whereas the other entities require contextual cues.
- 5.3 Results: Removing visual features causes more than a 5% performance drop in both datasets.Textual features make similar contributions; attention yields 1.5% gains on variable-template data but none on fixed-layout data.
- 5.4.1 Attention Analysis: Attention concentrates on a fixed address-containing anchor in VATI but emphasizes nearby, left, and upper segments plus slot keywords in variable-layout documents.The VATI pattern is associated with a shared template, while variable layouts reflect local document structure.
- 5.4.2 The Number of Graph Convolution Layers: Two graph-convolution layers are optimal for the task: additional layers do not help simple tasks, improve difficult tasks, and three layers overfit.The paper notes that the ideal layer count may need to adapt to the specific task.
- 5.4.3 Multi-Task Learning: Multi-task learning with segment classification benefits the extraction task on both datasets and helps training converge much faster.The two tasks are described as complementary, and the authors associate the approach with better generalization performance.
6 Conclusions and Future Works
The paper addresses entity extraction from visually rich documents with a graph convolution architecture that uses visual context, and reports experiments on two real-world datasets. It also identifies extending the framework to other VRD tasks as future work.
- A graph convolution architecture encodes text embeddings given visually rich context, followed by BiLSTM-CRF extraction.
- Experiments and analysis were conducted on two manually annotated real-world VRD datasets.
- The system outperforms BiLSTM baselines in the reported experiments.
- Future work will extend the graph convolution framework to other VRD tasks, including document classification.