Source-linked AI summary
Towards Robust Visual Information Extraction in Real World: New Dataset and Novel Solution
Jiapeng Wang, Chongyu Liu, Lianwen Jin, Guozhi Tang, Jiaxin Zhang, Shuaitao Zhang, Qianying Wang, Yaqiang Wu, Mingxiang Cai
TL;DR
Prior VIE systems split text spotting and information extraction into independent tasks, while Chinese-document benchmarks lacked comprehensive OCR and VIE annotations. VIES jointly performs detection, recognition, and extraction; it outperforms state-of-the-art methods on EPHOIE and reports a 9.01% F-score gain on SROIE under end-to-end information extraction.
Problem
Prior VIE systems separated text spotting and information extraction despite their high correlation, and existing benchmarks left Chinese OCR and VIE coverage limited.
Source: Introduction · p. 1, Datasets for Visual Information… · p. 2Method
VIES is an end-to-end trainable framework that jointly detects, recognizes, and extracts information, using visual and semantic features with reciprocal semantic guidance.
Source: Introduction · p. 1Results
VIES achieves superior performance on EPHOIE and a 9.01% F-score gain over previous state-of-the-art methods on SROIE under end-to-end information extraction.
Source: Conclusion · p. 7Takeaways & Limitations
EPHOIE provides a fully annotated first Chinese benchmark for both OCR and VIE.
Source: Conclusion · p. 7VIE remains an open problem, with complex layouts and backgrounds, reliance on complete annotations, and accumulating errors among issues not well addressed.
Source: Conclusion · p. 7
Abstract
from arXiv · showhide
Visual information extraction (VIE) has attracted considerable attention recently owing to its various advanced applications such as document understanding, automatic marking and intelligent education. Most existing works decoupled this problem into several independent sub-tasks of text spotting (text detection and recognition) and information extraction, which completely ignored the high correlation among them during optimization. In this paper, we propose a robust visual information extraction system (VIES) towards real-world scenarios, which is a unified end-to-end trainable framework for simultaneous text detection, recognition and information extraction by taking a single document image as input and outputting the structured information. Specifically, the information extraction branch collects abundant visual and semantic representations from text spotting for multimodal feature fusion and conversely, provides higher-level semantic clues to contribute to the optimization of text spotting. Moreover, regarding the shortage of public benchmarks, we construct a fully-annotated dataset called EPHOIE (https://github.com/HCIILAB/EPHOIE), which is the first Chinese benchmark for both text spotting and visual information extraction. EPHOIE consists of 1,494 images of examination paper head with complex layouts and background, including a total of 15,771 Chinese handwritten or printed text instances. Compared with the state-of-the-art methods, our VIES shows significant superior performance on the EPHOIE dataset and achieves a 9.01% F-score gain on the widely used SROIE dataset under the end-to-end scenario.
Introduction
The paper identifies disconnected text spotting and information extraction as a source of redundant computation and weak cross-module optimization. It proposes VIES, an end-to-end framework, and EPHOIE, a Chinese benchmark designed for complex layouts and handwritten text.
Introduction
- Earlier approaches separated text spotting from information extraction, redundantly reconstructing features and preventing the modules from jointly optimizing toward the task objective.The paper motivates a benchmark for real-world documents because existing resources did not adequately cover complex layouts and handwritten text.Source: Introduction · pp. 1–2
- VIES jointly trains text detection, recognition, and information extraction, sharing visual and semantic features and feeding higher-level semantic clues back to text spotting.Its vision and semantics coordination mechanisms connect the text-spotting branches with information extraction, while AFFM fuses visual, semantic, and location features at segment and token levels.Source: Introduction · pp. 1–2
- EPHOIE is a fully annotated Chinese benchmark for text spotting and visual information extraction, with 1,494 examination-paper images and 15,771 text instances.The images include handwritten and printed Chinese text, complex layouts, and noisy backgrounds.Source: Introduction · p. 2
- VIES outperforms state-of-the-art methods on EPHOIE and widely used benchmarks, according to the reported experiments.The conclusion reports a 9.01% F-score gain over previous state-of-the-art methods on SROIE under end-to-end information extraction.Source: Introduction · p. 2
Ralated Work
Prior VIE datasets and methods leave gaps in coverage of real-world Chinese documents and in joint treatment of OCR and information extraction. The related work contrasts these datasets with OCR pipelines, multimodal extraction methods, and end-to-end systems.
Datasets for Visual Information Extraction
- EPHOIE addresses demand for a Chinese dataset spanning OCR and VIE with real documents, varied layouts, and diverse text styles.SROIE focuses on printed-English scanned receipts, while a prior Chinese benchmark used fixed layouts, mostly synthetic images, and no OCR annotations.Source: Datasets for Visual Information… · p. 2
Methods for Visual Information Extraction
- Earlier pipelines converted text-spotting results into plain text before sequence labeling, but often lost visual and location information.Rule-based and template-matching approaches were also reported to generalize poorly.Source: Methods for Visual Information… · p. 2
- Later extraction methods incorporated visual and spatial cues through bounding boxes, document-layout modeling, graph embeddings, or CNNs, but traditional methods still ignored OCR.Examples include Post-OCR parsing, LayoutLM (Xu et al. 2020), GraphIE (Qian et al. 2019), PICK (Yu et al. 2020), and CharGrid (Katti et al. 2018).Source: Methods for Visual Information… · p. 2
- End-to-end approaches had limitations including fixed-layout constraints, difficulty with mixed-category text segments, or reliance on relatively simple OCR scenarios.These limitations are attributed respectively to (Guo et al. 2019), (Carbonell et al. 2020), and (Zhang et al. 2020).Source: Methods for Visual Information… · p. 2
Examination Paper Head Dataset for OCR
EPHOIE is introduced as a Chinese benchmark for OCR and visual information extraction, built from real examination-paper heads. Its annotations cover text geometry and content as well as entity key-value information.
and Information Extraction
- Table 1 compares EPHOIE with SROIE, using H and Q to denote horizontal and arbitrary quadrilateral text.Source: Table 1 · p. 3
Dataset Description
- EPHOIE contains 1,494 real examination-paper images and 15,771 annotated text instances, with 1,183 images for training and 311 for testing.The benchmark includes handwritten and printed characters; the images are cropped to paper-head regions containing key information.Source: Dataset Description · p. 3
Annotation Details
- Annotations use four vertices for text boxes and record text content, key-value links, and entity categories, including multiple entities within a segment.Source: Annotation Details · p. 3
Methodology
VIES unifies text detection, recognition, and information extraction around a shared backbone. Coordination mechanisms pass visual and semantic information between branches, while adaptive fusion combines features for entity labeling.
Methodology
- A shared backbone feeds detection, recognition, and information extraction; text-spotting features and spatial box cues are fused at segment and token levels before entity labeling.Source: Methodology · p. 3
Text Detection with Vision Coordination Mechanism (VCM)
- VCM transfers visual features from detection to information extraction and returns supervisory feedback that contributes to detection optimization.The visual features include document cues such as shapes, fonts, and colors; IE gradients can help detection learn more general representations.Source: Text Detection with Vision… · p. 3
Text Recognition with Semantics Coordination Mechanism (SCM)
- SCM sends token-level semantic representations from recognition to IE, while higher-level IE constraints guide recognition training.Source: Text Recognition with Semantics… · p. 4
- AFFM combines visual, semantic, and spatial features at segment and token levels, then re-couples local token and global segment information for labeling.Token-level visual and spatial features are generated from boxes divided along each text segment’s longest side.Source: Text Recognition with Semantics… · p. 5
Sequence Labeling
- A BiLSTM encodes the re-coupled feature sequence, and a fully connected layer followed by a CRF learns entity-sequence constraints.Source: Sequence Labeling · p. 5
Optimization Strategy
- Training uses a weighted sum of detection, recognition, and IE losses, with separate hyperparameters controlling the detection and recognition loss tradeoffs.Detection loss covers text classification, box regression, and mask identification; recognition uses cross-entropy, and IE uses CRF loss.Source: Optimization Strategy · p. 5
Implement Details
The implementation combines established detection and recognition components with specified convolution and attention configurations, then trains the spotting branches before end-to-end optimization.
- Text detection uses Mask R-CNN with a ResNet-50 backbone and FPN, while recognition uses an LSTM attention mechanism.Source: Implement Details · p. 5
- SCM uses 1D convolutions of sizes 2, 3, and 4; AFFM uses 4 heads, 3 sublayers, and feature dimensions of 256 and 512.Source: Implement Details · p. 5
- Both loss tradeoff hyperparameters are 1.0; end-to-end training follows spotting-branch pretraining and uses ADADELTA with branch learning rates 0.1 and 1.0.The learning rates are reduced by a tenth every 25 epochs, twice.Source: Implement Details · p. 5
Ablation Study
The ablations evaluate end-to-end optimization and the VCM, SCM, and AFFM feature-fusion design. They report gains from the proposed coordination and multimodal representations, with semantic, visual, and spatial features contributing differently.
Ablation Study
- The ablation study evaluates the influences of multiple framework components on EPHOIE.The end-to-end optimization comparison varies whether IE gradients reach text spotting and pairs optimization approaches with IE structures.Source: Ablation Study · p. 5, Effect of End-to-End Optimization · p. 5
Effect of End-to-End Optimization
- VIES outperforms four counterparts across detection, recognition, and extraction; TRIE improves some counts but reduces high-IoU detection and recognition.E2E(Ours) + IE stage in TRIE is comparable at low IoU, while VIES gains across tasks over that configuration and E2E(Ours) + GAT.Source: Effect of End-to-End Optimization · pp. 5–6, Table 2 · p. 6
Effects of VCM and SCM
- VCM and SCM's specific designs maximize end-to-end optimization benefits; simply combining text-spotting and IE branches is insufficient.The experiments compare alternative structures for VCM and SCM.Source: Table 3 · p. 6, Effects of VCM and SCM · p. 6
Effect of multi-source features
- Setting (4)'s multimodal fusion performs best; semantics alone is satisfactory, visual cues add significant gains, and spatial features slightly improve over semantics alone.AFFM lets features model inter- and intra-segment relations and can mitigate negative effects from errors across sources.Source: Table 4 · p. 6, Effect of multi-source features · p. 6
Comparison with the State-of-the-Arts
The comparisons report VIES results on EPHOIE and SROIE against state-of-the-art methods. On SROIE, the reported end-to-end improvement is especially large, while the receipt setting is relatively simple compared with EPHOIE.
Comparison with the State-of-the-Arts
- The comparison re-implements state-of-the-art methods from original papers or available source code.Source: Comparison with the State-of-the-Arts · p. 6
Results on EPHOIE Dataset
- VIES is superior on EPHOIE, where multimodal methods outperform counterparts; its end-to-end pipeline reduces negative effects from less accurate text spotting.Figure 5 shows prediction examples, with different colors denoting different entities.Source: Results on EPHOIE Dataset · pp. 6–7, Table 5 · p. 7, Figure 5 · p. 7
Results on SROIE Dataset
- From 82.06 to 91.07 under End-to-End SROIE, VIES surpasses prior state-of-the-art results and achieves competitive results with Ground Truth inputs.The SROIE receipt layout, fonts, and background are relatively simple compared with EPHOIE.Source: Results on SROIE Dataset · p. 7
Conclusion
The conclusion presents VIES as an end-to-end system for detection, recognition, and information extraction, alongside the EPHOIE benchmark. It reports strong benchmark results and identifies unresolved challenges for visual information extraction.
Conclusion
- VIES is an end-to-end trainable framework for simultaneous text detection, recognition, and information extraction.Source: Conclusion · p. 7
- 9.01% F-score gains over previous state-of-the-art methods on SROIE under end-to-end information extraction; VIES also achieves superior performance on EPHOIE.Source: Conclusion · p. 7
- EPHOIE is a fully annotated first Chinese benchmark for both OCR and visual information extraction.Source: Conclusion · p. 7
- Complex layouts and backgrounds, reliance on complete annotations, and accumulated errors remain unresolved challenges for visual information extraction.Source: Conclusion · p. 7