Source-linked AI summary
TRIE: End-to-End Text Reading and Information Extraction for Document Understanding
Peng Zhang, Yunlu Xu, Zhanzhan Cheng, Shiliang Pu, Jing Lu, Liang Qiao, Yi Niu, Fei Wu
TL;DR
Existing document-understanding pipelines separate text reading from information extraction, despite their mutual correlation. TRIE unifies the tasks so multimodal text-reading features support extraction and extraction semantics contribute to reading optimization. Across three diverse real-world benchmarks, it reports better efficiency and accuracy than state-of-the-art methods.
Problem
Conventional document-understanding methods separate text reading from information extraction, leaving their mutual correlation unmodeled.
Source: §1 · p. 2Method
TRIE jointly trains text reading and information extraction, fusing multimodal visual and textual features for extraction and using extraction semantics to optimize reading.
Source: §1 · p. 2, §3.1 · p. 3Results
Takeaways & Limitations
The framework is presented as applicable across fixed and variable layouts and structured and semi-structured text types.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Since real-world ubiquitous documents (e.g., invoices, tickets, resumes and leaflets) contain rich information, automatic document image understanding has become a hot topic. Most existing works decouple the problem into two separate tasks, (1) text reading for detecting and recognizing texts in images and (2) information extraction for analyzing and extracting key elements from previously extracted plain text. However, they mainly focus on improving information extraction task, while neglecting the fact that text reading and information extraction are mutually correlated. In this paper, we propose a unified end-to-end text reading and information extraction network, where the two tasks can reinforce each other. Specifically, the multimodal visual and textual features of text reading are fused for information extraction and in turn, the semantics in information extraction contribute to the optimization of text reading. On three real-world datasets with diverse document images (from fixed layout to variable layout, from structured text to semi-structured text), our proposed method significantly outperforms the state-of-the-art methods in both efficiency and accuracy.
1 INTRODUCTION
Visually rich documents combine text and visual information, but conventional pipelines separate text reading from information extraction and discard some of that context. TRIE unifies the tasks, allowing multimodal features and IE semantics to reinforce one another; it reports superior efficiency and accuracy on three diverse real-world benchmarks.
1 INTRODUCTION
- The paper's VRD scope includes both text and vision, with information such as layout, tabular structure, font size, and font color.The introduction notes that recognizing text and extracting valuable contents from VRDs can facilitate information entry, retrieval, and compliance checks.Source: §1 · pp. 1–2
- Existing approaches separate text reading from IE: OCR lacks IE semantic supervision, while IE typically uses serialized text without rich visual information.Prior work also leaves the tasks' correlation underexplored and incurs redundant computation and time cost through stagewise training.Source: §1 · p. 2
- TRIE bridges text reading and IE with shared position, visual, and textual features so the tasks can reinforce each other.Text-reading features are fused for IE, while IE semantics contribute to optimizing text reading.Source: §1 · p. 2
- The framework is presented as end-to-end trainable and able to handle fixed and variable layouts and structured and semi-structured text.The authors describe it as the first such end-to-end trainable VRD framework, to the best of their knowledge.Source: §1 · p. 2
- TRIE significantly outperforms state-of-the-art methods in efficiency and accuracy on three real-world benchmarks spanning fixed and variable layouts and structured and semi-structured text.The benchmarks cover diverse document images.Source: §1 · p. 2
2 RELATED WORKS
Related work moves from text-reading methods that jointly detect and recognize text to IE methods that increasingly incorporate document layout and visual information. The reviewed approaches nevertheless leave opportunities for modeling the relationship between reading and extraction and for handling varied document layouts and text types.
2.1 Text Reading
- Joint text detection and recognition methods capture relations between those subtasks, inspiring investigation of the broader relationship between text reading and IE.The paper positions its work as the first to focus on relations between text reading and information extraction.Source: §2.1 · p. 3
- Text-reading research had already explored end-to-end joint detection and recognition, but the paper identifies reading–IE relations as a less-explored connection.The cited joint reading work is described as capturing relations within detection and recognition subtasks.Source: §2.1 · p. 3
2.2 Information Extraction
- Earlier IE methods used rules and templates, while later learning-based methods added layout, position, and image embeddings but still focused on IE.Rule-based approaches relied on predefined rules and did not generalize across document templates; plain-text serialization also discarded layout information.Source: §2.2 · p. 3
- Prior VRD IE approaches had document-type constraints, whereas TRIE is presented as handling fixed and variable layouts and structured and semi-structured text.The reviewed concurrent approaches are described as limited to fixed layouts and one text type, or to simple layouts and non-structured text.Source: §2.2 · p. 3
3 METHODOLOGY
The methodology section is organized around the overall architecture, three module descriptions, and optimization functions.
3 METHODOLOGY
- The methodology first introduces TRIE's overall architecture.The roadmap locates this in Section 3.1.Source: §3 · p. 3
- The next three sections describe the text reading module, multimodal context block, and information extraction module, respectively.Source: §3 · p. 3
- The methodology then presents the optimization functions in Section 3.5.Source: §3 · p. 3
3.1 Overall Architecture
TRIE combines text reading, a multimodal context block, and information extraction in one jointly trained architecture. The context combines visual and textual information, and information can flow in both directions between reading and extraction.
3.1 Overall Architecture
- TRIE has a text-reading module, a multimodal context block, and an information-extraction module.Text reading localizes and recognizes text, while information extraction identifies entities of interest.Source: §3.1 · p. 3
- The context block combines visual context summarizing local text patterns with textual context modeling relations across the document.The fused context and textual features are used to extract entities.Source: §3.1 · p. 3
- Joint end-to-end training allows bidirectional information flow, with text-reading features supporting IE and IE loss supervising text-reading optimization.Source: §3.1 · p. 3
3.2 Text Reading Module
The text reading module detects text regions, recognizes their contents, and supplies visual, textual, and positional features for downstream processing. In the overall architecture, text regions, content, and entities are predicted in one forward pass.
3.2 Text Reading Module
- A shared backbone feeds a text-detection branch that predicts the locations of possible text regions.The paper uses ResNet and FPN as the backbone.Source: §3.2 · p. 3
- The overall network predicts text regions, text content, and entities of interest in a single forward pass.Source: Figure 3 · p. 4
- For each text region, RoIAlign extracts region features that an encoder and attention-based decoder use to generate its character sequence.During testing, the decoder uses the character predicted at the previous step.Source: §3.2 · p. 4
- The text reading module outputs text positions, visual features, and textual features for the detected texts.These outputs are denoted B, C, and Z, respectively.Source: §3.2 · p. 4
3.3 Multimodal Context Block
The multimodal context block combines complementary visual and textual context with text position information for entity extraction. Visual context captures local patterns, while textual context models long-distance relationships between texts.
3.3 Multimodal Context Block
- The multimodal context block brings together position, visual, and textual features, whose visual and textual contexts are complementary in information extraction.Source: §3.3 · p. 4
3.3.1 Visual Context.
- The visual context reuses text-region features from the reading module, whose deep backbone and FPN lateral connections summarize local visual patterns.Source: §3.3.1 · p. 4
3.3.2 Textural Context.
- Textual context models fine-grained long-distance dependencies by applying self-attention to text embeddings that combine position embeddings with aggregated character semantics.The attention supports a variable number of texts.Source: §3.3.2 · pp. 4–5
3.4 Information Extraction Module
The information extraction module uses visual and textual context to distinguish entities, while character-level textual features support entity extraction. It adaptively fuses the contexts before modeling character dependencies and predicting labels.
3.4 Information Extraction Module
- Visual and textual context help distinguish entities, while character-level textual features provide semantic information for entity extraction.Source: §3.4 · p. 5
3.4.1 Adaptive Multi-modality Context Fusion.
- For each text, spatially averaged visual context and textual context are combined as a weighted sum, with automatically learnable weights.The context modalities are projected to the same dimension before fusion.Source: §3.4.1 · p. 5
3.4.2 Entity Extraction.
- The fused context is concatenated to each character feature, then a Bidirectional-LSTM models character dependencies before a fully connected layer projects outputs to the IOB label space.Source: §3.4.2 · p. 5
3.5 Optimization
The network is trained end-to-end with losses from text detection, recognition, and information extraction. Shared visual and textual features connect the tasks, allowing information extraction semantics to contribute to optimizing text reading.
3.5 Optimization
- The end-to-end network's losses come from text detection, text recognition, and information extraction.Source: §3.5 · p. 5
- The recognition and information extraction losses use their respective ground-truth labels, while λ_recog and λ_info control the trade-off between losses.Source: §3.5 · p. 5
- Shared visual and textual features connect text reading and information extraction, whose semantic feedback contributes to optimizing the shared convolutions and text reading module.Source: §3.5 · p. 5
4 EXPERIMENTS
Experiments evaluate TRIE on three datasets spanning fixed and variable layouts and structured and semi-structured text. The reported results favor TRIE across the benchmarks, with a noted annotation-related exception for one Taxi Invoices entity.
4.1 Datasets
- The three benchmarks are SROIE, Taxi Invoices, and Resumes, spanning fixed to variable layouts and structured to semi-structured text.Taxi Invoices has 5,000 images and nine entity types; SROIE has 626 training and 347 test receipts; Resumes has 2,475 images and six entity types.Source: §4.1 · p. 6
4.2.1 Network Details:
- The model uses a ResNet-50/FPN backbone, Faster R-CNN text detection, LSTM-attention recognition, and convolutional and attention modules for information extraction.The reported hidden dimensions are 256 for recognition and text embeddings and 128 for the entity-extraction BiLSTM.Source: §4.2.1 · p. 6
4.2.2 Training:
- TRIE is trained end-to-end for 150 epochs with batch size two images per GPU, while comparison pipelines train reading and extraction independently to convergence.Experiments use ADADELTA and two NVIDIA Tesla V100 GPUs.Source: §4.2.2 · p. 6
4.3.1 Network Settings of SOTA counterparts and Evaluation Metric.
- Pipeline baselines pair text reading with Chargrid, LSTM-CRF, or GCN, and experiments use F1-score for evaluation.The comparison methods are reimplemented with their respective network settings.Source: §4.3.1 · p. 6
4.3.2 Results.
- TRIE significantly outperforms the Taxi Invoices counterparts except on Pick-up time, where blurred values labeled NULL lower its score despite correct extraction.The paper attributes the exception to annotations such as the blurred time “18:47” being tagged NULL.Source: §4.3.2 · p. 6, Table 3 · p. 7, Figure 4 · p. 7
- On SROIE, TRIE beats three counterparts by a large margin in Setting 1 and achieves competitive performance against stronger comparison methods in Setting 2.Setting 1 trains the text-reading module independently; Setting 2 uses official text boxes and transcripts for leaderboard comparisons.Source: §4.3.2 · pp. 6–7, Table 4 · p. 7
- On variable-layout, semi-structured Resumes, TRIE gains comprehensively by combining context features with character-level entity extraction.The discussion contrasts its results with GCN’s difficulty on flexible layouts and the different entity-level strengths of Chargrid and NER.Source: Table 5 · p. 7, §4.3.2 · p. 7
4.4 Discussion
The discussion examines multimodal features, end-to-end training, speed, and context-block capacity. Its findings favor combining visual and textual context, while layer and head settings vary with dataset complexity.
4.4.1 Effect of multi-modality features on information extrac-
- Fusing visual and textual context gives the best Taxi Invoices result; either context alone improves on text-reading features alone, and visual context performs slightly better than textual context.The accompanying figure illustrates that visual cues can help with distinctive Taxi Code formatting, while visual context may fail on unusual invoice layouts.Source: §4.4.1 · p. 7, Figure 5 · p. 8
4.4.2 Effect of the end-to-end framework on text reading:
- End-to-end training yields a 0.9 F1-score gain for GCN over separately trained text reading, and the paper reports that joint modeling benefits both tasks.Examples include correcting the recognition of “23.3km” and “2017-12-21” compared with pipeline training.Source: Table 7 · p. 8, §4.4.2 · p. 8
4.4.3 Speed:
- TRIE runs at 1.76 fps on Resumes, faster than the full Chargrid(TR), NER(TR), and GCN(TR) pipelines at 1.13, 1.69, and 1.62 fps.The paper attributes the speed advantage to feature sharing; text reading is the main time cost at 1.83 fps.Source: §4.4.3 · p. 8
4.4.4 Ablation on layers and heads in Textural Context Block:
- The best Textural Context Block depth and head count depend on dataset complexity: simpler fixed-layout Taxi Invoices favor smaller settings, while variable-layout Resumes require deeper settings.The paper warns that deeper layers and more heads can lead to overfitting on Taxi Invoices.Source: Table 8 · p. 8, §4.4.4 · p. 8
5 CONCLUSION
TRIE jointly trains text reading and information extraction so the tasks can reinforce each other. Across benchmarks spanning multiple layout and text types, the paper reports gains in accuracy and efficiency.
5 CONCLUSION
- TRIE bridges text reading and information extraction in an end-to-end network for document understanding.The two tasks are jointly trained and can mutually reinforce each other.Source: §5 · p. 8
- Visual and textual features from text reading boost information extraction, while information-extraction loss also supervises text-reading optimization.Source: §5 · p. 8
- Across benchmarks with structured to semi-structured text and fixed to variable layouts, TRIE significantly outperforms three state-of-the-art methods in efficiency and accuracy.Source: §5 · p. 8