Source-linked AI summary
Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solution
Jianfeng Kuang, Wei Hua, Dingkang Liang, Mingkun Yang, Deqiang Jiang, Bo Ren, Xiang Bai
TL;DR
Existing VIE datasets lack the image and entity diversity needed to represent real-world challenges. The paper introduces the camera-image POIE dataset and an end-to-end framework using contrastive learning to connect OCR with information extraction. Experiments show performance gains on both POIE and SROIE, while varied forms of the same entity remain a challenge.
Problem
Existing VIE datasets mainly contain document images with insufficient diversity in layouts, backgrounds, and entity categories to represent real-world challenges.
Source: §1 · pp. 2–3Method
The paper introduces the POIE camera-image dataset and an end-to-end VIE framework with CFAM, which uses contrastive learning to connect OCR and information extraction.
Source: §1 · pp. 3–4Results
The proposed framework achieves performance gains on both SROIE and POIE.
Source: §6 · p. 15Takeaways & Limitations
The experiments support POIE as a more practical dataset for promoting advanced VIE algorithms.
Source: §6 · p. 15Different forms of the same entity remain a challenge and continue to affect information-extraction performance.
Source: §5.5 · p. 14
Abstract
from arXiv · showhide
Visual information extraction (VIE), which aims to simultaneously perform OCR and information extraction in a unified framework, has drawn increasing attention due to its essential role in various applications like understanding receipts, goods, and traffic signs. However, as existing benchmark datasets for VIE mainly consist of document images without the adequate diversity of layout structures, background disturbs, and entity categories, they cannot fully reveal the challenges of real-world applications. In this paper, we propose a large-scale dataset consisting of camera images for VIE, which contains not only the larger variance of layout, backgrounds, and fonts but also much more types of entities. Besides, we propose a novel framework for end-to-end VIE that combines the stages of OCR and information extraction in an end-to-end learning fashion. Different from the previous end-to-end approaches that directly adopt OCR features as the input of an information extraction module, we propose to use contrastive learning to narrow the semantic gap caused by the difference between the tasks of OCR and information extraction. We evaluate the existing end-to-end methods for VIE on the proposed dataset and observe that the performance of these methods has a distinguishable drop from SROIE (a widely used English dataset) to our proposed dataset due to the larger variance of layout and entities. These results demonstrate our dataset is more practical for promoting advanced VIE algorithms. In addition, experiments demonstrate that the proposed VIE method consistently achieves the obvious performance gains on the proposed and SROIE datasets.
1 Introduction
Existing VIE datasets do not adequately represent the layout, background, and entity variation found in real-world images. The paper introduces POIE and an end-to-end framework using contrastive learning to connect OCR and information extraction.
1 Introduction
- Existing VIE datasets mainly contain moderate-scale document images and cannot fully reveal real-world challenges such as varied layouts, backgrounds, fonts, and entities.These conditions make VIE in the wild more difficult.Source: §1 · pp. 2–3
- POIE contains 3,000 camera images and 111,155 text instances, with diverse layouts and distortions and up to 21 entity types.The paper describes POIE as over 2× larger than the previous largest public VIE dataset.Source: §1 · p. 3
- Prior end-to-end methods perform worse on POIE, especially in information extraction, and directly pass OCR features to the extraction module.The authors attribute this to a semantic gap between OCR and information extraction.Source: §1 · p. 3
- CFAM uses contrastive learning to connect OCR instance features with entity features and adjust instance features for information extraction.The module constructs a similarity matrix between entity and instance features.Source: §1 · p. 4
2 Related Works
Prior VIE work includes document-focused datasets, two-stage and end-to-end methods, and contrastive-learning approaches from other fields. The paper positions its framework as connecting OCR and information extraction through contrastive learning.
2.1 VIE Datasets
- Existing VIE datasets are mainly moderate-scale document images, while the paper calls for greater layout, background, and entity diversity.SROIE contains scanned English receipts; EPHOIE targets Chinese document VIE.Source: §2.1 · p. 4
2.2 VIE Methods
- Two-stage methods model OCR outputs for extraction but can inherit OCR errors; end-to-end methods vary in layout flexibility, cost, or pretraining needs.EATEN is limited to fixed layouts, while Donut needs massive pretraining data; previous end-to-end methods use OCR features as extraction inputs.Source: §2.2 · pp. 4–5
2.3 Contrastive Learning
- Contrastive learning brings positive pairs together and separates negative pairs, but had not been thoroughly studied for VIE.The paper uses it to supervise relations between OCR instance features and entity features.Source: §2.3 · p. 5
3 POIE Dataset
POIE is collected from real-world product images and manually annotated. Its varied image conditions and entity types are intended to represent challenges of VIE in the wild, and it is evaluated on detection, recognition, and information extraction.
3.1 Data Collection
- POIE contains 3,000 manually checked product images with 111,155 text instances after low-quality and blurred images are discarded.Eight experienced labelers checked locations, transcriptions, and entity values, using Azure and Baidu OCR for pre-labeling.Source: §3.1 · pp. 5–6
3.2 Data Characteristics
- POIE includes diverse image styles, noisy backgrounds, distortions, and 21 entity types, with some entities appearing in different forms.Entities may contain multiple words and appear zero or once in an image; the authors say these properties can help improve model robustness and generalization.Source: §3.2 · p. 6
3.3 Data Split and Evaluation Protocol
- POIE is split into 2,250 training images and 750 test images, evaluated with F1-Score for detection, recognition, and information extraction.The protocol follows SROIE and EPHOIE.Source: §3.3 · p. 6
4 Our Method
The end-to-end framework combines text detection, recognition, contrast-guided feature adjustment, and information extraction. CFAM connects OCR and entity representations, and four weighted losses train the system jointly.
4 Our Method
- The end-to-end pipeline detects text, processes instance features through CFAM and recognition, then combines recognition, encoded, and similarity features for structural prediction.The information extraction module uses LSTM-CRF to predict entity categories at the character level.Source: §4 · pp. 6–7, §4.2 · p. 8
4.1 Contrast-guided Feature Adjustment Module
- CFAM contextualizes instance features, combines encoded features with learnable entity embeddings, and computes a similarity matrix supervised by annotated entity correspondences.Three self-attention layers generate encoded features that connect instances and transfer visual features; encoded features and learnable embeddings jointly form entity features.Source: §4.1 · pp. 7–8
4.2 Information Extraction Module
- The information extraction module uses recognition, encoded, and similarity features as visual, textual, and correlative signals to predict character-level entity categories with LSTM-CRF.It outputs the final structured results.Source: §4.2 · p. 8
4.3 Loss Function
- End-to-end training uses a weighted sum of detection, recognition, information-extraction, and contrastive-learning losses, with hyperparameters controlling their tradeoff.Focal loss is used for contrastive learning because entity counts are imbalanced.Source: §4.3 · p. 9
5 Experiments
Experiments compare POIE with SROIE and report state-of-the-art results for the proposed method on both datasets. Ablations examine CFAM, its entity features, and its transfer to other methods.
5.1 Implementation Details
- Training used four TITAN Xp GPUs, batch size four, and Adadelta; the training schedule was 600 epochs for SROIE and 200 for POIE.The authors state these epoch counts were set for a fair comparison.Source: §5.1 · p. 10
5.2 Analysis of Our Proposed Dataset
- Pure IE scores fall from 96.12 to 82.91 for VIES and from 96.18 to 81.79 for TRIE from SROIE to POIE.Text spotting performance differs only slightly between the datasets; the authors attribute the pure-IE drop to variable layouts and entities. Table 2 presents the comparison.Source: §5.2 · p. 11
- End-to-end TRIE and VIES scores fall from 82.06 to 76.37 and from 83.44 to 77.19, respectively, from SROIE to POIE.The authors interpret this as further evidence that POIE is practical for evaluating end-to-end methods.Source: §5.2 · p. 11
5.3 The Comparisons of Our Method and SOTA
- The proposed method achieves state-of-the-art results on POIE and SROIE, outperforming other methods by an obvious margin.The comparison focuses on information-extraction F1-score; previous SROIE results without post-processing are the main comparison.Source: Table 3 · p. 11, §5.3 · p. 12
5.4 Ablation Studies
- With CFAM, POIE reaches 97.89% in DET, 91.68% in REC, and 79.54% in IE; contrastive loss Lc also promotes performance.Direct classification improves IE but negatively affects detection and recognition; the authors attribute CFAM’s results to bridging relations between OCR and information extraction.Source: Table 4 · p. 12, §5.4 · p. 13, Table 5 · p. 14
- Combining learnable entity embeddings with encoded features performs best; either strategy alone produces lower results.Without encoded features, entity features are nearly alike and underuse instance information; without learnable embeddings, they rely entirely on encoded features.Source: §5.4 · p. 14, Table 6 · p. 14
- CFAM improves POIE TRIE and VIES by 2.45% and 1.92%, respectively, and also improves both methods on SROIE.The authors report this as evidence that CFAM generalizes to other methods.Source: §5.4 · p. 14, Table 7 · p. 15
5.5 Limitation
- Different forms of the same entity remain challenging and still affect information-extraction performance.The authors identify this as an unresolved limitation and say they will explore commonality between entity forms.Source: §5.5 · p. 14
6 Conclusion
The paper presents POIE, a large-scale English camera-image dataset intended to reflect real-world VIE challenges, and an end-to-end framework with a plug-and-play CFAM. Experiments report consistent performance gains on SROIE and POIE, and the authors identify dataset expansion as future work.
6 Conclusion
- Experiments show the proposed framework consistently achieves obvious performance gains on SROIE and POIE.The authors also conclude that POIE is more practical for promoting advanced VIE algorithms.Source: §6 · p. 15
- The authors conclude that POIE is more practical for promoting advanced VIE algorithms.POIE is a large-scale English dataset of camera images intended to reflect real-world challenges.Source: §6 · p. 15
- The proposed end-to-end framework uses a plug-and-play CFAM and contrastive learning to narrow the semantic gap between OCR and information extraction.The framework also designs task representations for contrastive learning.Source: §6 · p. 15
- Future work will extend POIE with more images, entities, and diverse natural-scene disturbances.Source: §6 · p. 15