Source-linked AI summary
ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction
Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shjian Lu, C. V. Jawahar
TL;DR
SROIE targets the under-researched challenge of recognizing and extracting information from scanned receipts, which are important for document-intensive applications. The ICDAR 2019 competition addresses this gap with three tasks, a 1000-image annotated dataset, and evaluation protocols; participation was broad, while receipt OCR and information extraction remained challenging.
Problem
Scanned receipt OCR and key information extraction are commercially important but face difficult image conditions, stringent accuracy requirements, heavy manual work, and limited dedicated research.
Method
The authors organize the ICDAR 2019 SROIE competition around text localisation, receipt OCR, and key information extraction, supported by a new annotated dataset and evaluation protocols.
Results
16 teams exceeded 90% Hmean for Task 1 and 7 exceeded 90% for Task 2, yet even the best Task 2 OCR method did not meet the 99% accuracy requirement for receipt applications.
Takeaways & Limitations
The competition attracted broad interest and new approaches, while its results indicate that scanned receipt OCR and especially key information extraction require further research.
Abstract
from arXiv · showhide
Scanned receipts OCR and key information extraction (SROIE) represent the processeses of recognizing text from scanned receipts and extracting key texts from them and save the extracted tests to structured documents. SROIE plays critical roles for many document analysis applications and holds great commercial potentials, but very little research works and advances have been published in this area. In recognition of the technical challenges, importance and huge commercial potentials of SROIE, we organized the ICDAR 2019 competition on SROIE. In this competition, we set up three tasks, namely, Scanned Receipt Text Localisation (Task 1), Scanned Receipt OCR (Task 2) and Key Information Extraction from Scanned Receipts (Task 3). A new dataset with 1000 whole scanned receipt images and annotations is created for the competition. In this report we will presents the motivation, competition datasets, task definition, evaluation protocol, submission statistics, performance of submitted methods and results analysis.
I. INTRODUCTION
SROIE addresses challenging, commercially important receipt OCR and key information extraction tasks that remain underrepresented in research. The ICDAR 2019 competition introduces a dataset, three focused tasks, and evaluation procedures to stimulate progress.
- Motivation: SROIE recognizes receipt text and extracts key texts into structured documents for archiving, indexing, analytics, and office automation.Receipt OCR and information extraction support document-intensive financial, accounting, and taxation processes.
- Challenges: Low-quality scans and receipts’ stringent commercial accuracy requirements leave substantial manual work in existing SROIE systems.Challenges include poor paper, ink, and printing quality, scanning distortion, interfering text, long texts, and small fonts.
- Research gap: Despite commercial interest and related research in reading, layout analysis, and named entity recognition, few studies and prior competitions directly address SROIE.The authors identify a gap specific to scanned receipts and their information extraction requirements.
- Contributions: The competition provides a large-scale, well-annotated receipt dataset with challenging image conditions and blurred sensitive fields.The dataset is intended to complement existing OCR datasets while addressing privacy concerns.
- Contributions: Three tasks cover receipt text localisation, receipt OCR, and key information extraction, for which conventional named entity recognition approaches are not directly applicable.The competition also develops comprehensive evaluation methods for the tasks.
- Participation: The competition received 29, 24, and 18 valid submissions for Tasks 1, 2, and 3, respectively, indicating broad participation.The report analyzes motivation, datasets, task definitions, evaluation, submissions, methods, and results.
II. DATASET AND ANNOTATIONS
The SROIE dataset contains 1000 scanned receipts with task-specific annotations. It separates 600 annotated trainval images from 400 test images and records text locations, transcripts, and extraction labels.
- Dataset: The dataset contains 1000 whole scanned receipt images used across all three tasks with different annotations.Each receipt contains approximately four key text fields, and annotations mainly comprise digits and English characters.
- Dataset split: The trainval split provides 600 receipt images with annotations, while the test split contains 400 images released before the deadline.The split supports participant training and held-out evaluation.
- Annotations: Detection and OCR annotations include four-vertex rectangular text bounding boxes and the transcript for each text box.Locations use clockwise vertex ordering, and annotation files share image filenames.
- Annotations: Figure 1 illustrates the annotation formats used for the competition tasks.The figure presents examples for the task-specific annotation organization.
III. COMPETITION TASKS
The competition’s text localisation task asks participants to accurately locate text in receipt images using four-vertex annotations. It builds on conventional localisation tasks from earlier ICDAR competitions.
- Text localisation: Task 1 requires accurate localisation of receipt text at the level of four vertices, with ground truth at least at word level.Participants submit a ZIP file containing results for all test images.
2) Evaluation Protocol:
The evaluation protocol defines task-specific outputs and ranks localisation and OCR systems with F1-based measures. It accommodates differing detection granularities and repeated words.
- Text localisation: Text localisation uses a DetVal-based methodology to address one-to-many and many-to-one correspondences between detected texts.This supports evaluation when teams localize text at different levels, such as lines.
- Text localisation: Localisation ranking computes F1 from mean average precision and average recall.The protocol calculates mAP and average recall before deriving the final ranking score.
- Receipt OCR: Receipt OCR requires a list of recognized words without localization information and restricts words to Latin characters and numbers.The training ground truth is obtained by tokenizing transcription strings on spaces.
- Receipt OCR: OCR evaluation matches submitted words to ground-truth words, preserves repeated words, and uses F1 as the final ranking metric.Precision is based on correct matches among detected words, while recall is based on correct matches among ground-truth words.
C. Task 3- Key Information Extraction from Scanned Receipts
Task 3 requires extracting specified key fields from each receipt and submitting them in JSON format, with correctness determined by matching both content and category.
- Participants extract texts for a number of key receipt fields and save each receipt's results in a JSON file.
- An extraction is correct only when both its submitted content and category match the ground truth.
- F1 score, computed from mAP and recall, determines the ranking.
IV. ORGANIZATION
The competition used the RRC web portal and followed a staged schedule from website launch and registration through dataset release and Task 3 submission.
- The RRC web portal hosted competition information, dataset downloads, registration, and result submission.
- The website went online on February 15, 2019, and registration ran from February 20 to March 31, 2019.
- Training and validation datasets became available on March 1, 2019, while the test dataset became available on March 31, 2019.
- Task 3 submissions opened on April 23, 2019, and closed on May 5, 2019.
- By the deadline, the competition received 29 Task 1, 24 Task 2, and 18 Task 3 submissions, ranking only the last submission from teams with multiples.
V. SUBMISSION AND RESULTS
After submission deadlines, automated evaluation determined winners using each task's primary metric, while Task 1's leading method used iterative refinement and model ensembling.
- The RRC web team evaluated all submissions automatically using scripts after the submission deadlines.
- Winners for each task were determined by the score achieved on the corresponding primary metric.
- Figure 3 lists the top 10 Task 1 submissions, with the top three methods described in the report.
- The first-ranked Task 1 method, SCUT-DLVC-Lab-Refinement, used refinement-based Mask-RCNN, iterative removal of redundant information, and multi-backbone ensembling.
B. Task 2 Performance and Ranking
Task 2's leading methods combined modified recurrent OCR architectures, model ensembles, and synthetic receipt text with real-data fine-tuning.
- Figure 4 lists the top 10 Task 2 submissions, and the report describes the three highest-ranked methods.
- The first-ranked H&H Lab method used CRNN with a PVANet-like CNN, multiple GRU layers, and an adjusted training strategy.
- The second-ranked INTSIG-HeReceipt-Ensemble method trained models with varied backbones and recurrent structures, then ensembled them and split line outputs into words.
- The third-ranked Ping An method used an attention-based encoder-decoder, generated 2 millions of synthetic text lines, and fine-tuned on real receipt data.
C. Task 3 Performance and Ranking
Task 3 attracted diverse approaches, but key information extraction remained substantially less accurate than the other competition tasks. The results also indicate that strong performance on upstream detection and OCR was common among top Task 3 teams.
- Task 3 methods: Top-10 Task 3 submissions used varied approaches to address the new key information extraction problem.The report characterizes Task 3 as an open research issue and expects further innovative methods.
- Task 3 methods: Task 3’s first-ranking method lexicon-corrected recognition outputs and applied field-specific regular expressions to extract company, cash, date, and address information.The method built its lexicon from the training dataset and used different regular-expression patterns for each keyword.
- Task 3 performance: The competition paired information extraction with detection and OCR because those processes provide key inputs for extraction, and top Task 3 teams generally performed well on them.This task design was intended both to attract interest and to evaluate supporting capabilities.
- Task 3 performance: Only one Task 3 method exceeded 90% Hmean, reaching 90.49%, while more than half of submissions scored below 80%.These results show substantial remaining room for improvement in key information extraction.
VII. CONCLUSION
The competition established an early benchmark for scanned-receipt OCR and information extraction through new datasets, evaluation protocols, and three tasks. Its submissions showed broad interest, while key information extraction remained challenging and motivated larger, more demanding future benchmarks.
- Conclusion: The organizers created new datasets and evaluation protocols for three scanned-receipt competition tasks.The competition was among the first focused on OCR and information extraction for scanned receipts.
- Conclusion: A substantial number of submissions from academia and industry demonstrated broad interest in the topic.The conclusion reports participation across both academic and industrial communities.
- Conclusion: Submission performance indicated a large gap in expected information extraction performance, leaving the task highly challenging.The conclusion identifies key information extraction as relevant to other important document-analysis applications.
- Conclusion: Future extensions should use more challenging and larger datasets and applications.The competition’s new datasets were planned for release after the event.