Source-linked AI summary
ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVT
Yipeng Sun, Zihan Ni, Chee-Kheng Chng, Yuliang Liu, Canjie Luo, Chun Chet Ng, Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, Chee Seng Chan, Lianwen Jin
TL;DR
Existing scene-text datasets are costly to fully annotate and therefore relatively limited for training robust text-reading systems. The paper presents ICDAR 2019-LSVT, a large street-view text challenge using full and weak annotations, and summarizes its tasks, evaluation, participation, results, and remaining challenges.
Problem
Full annotation is costly, leaving scene-text training data relatively limited for optimizing deep learning algorithms.
Method
The competition introduces LSVT with fully annotated and weakly annotated street-view images for text detection and end-to-end text spotting.
Results
ICDAR 2019-LSVT received 132 valid submissions from 41 teams across its two tasks, while results analysis summarized system abilities and remaining challenges.
Takeaways & Limitations
LSVT provides a large-scale benchmark combining street-view text, full annotations, weak annotations, and a large testing set for studying robust text reading.
Abstract
from arXiv · showhide
Robust text reading from street view images provides valuable information for various applications. Performance improvement of existing methods in such a challenging scenario heavily relies on the amount of fully annotated training data, which is costly and in-efficient to obtain. To scale up the amount of training data while keeping the labeling procedure cost-effective, this competition introduces a new challenge on Large-scale Street View Text with Partial Labeling (LSVT), providing 50, 000 and 400, 000 images in full and weak annotations, respectively. This competition aims to explore the abilities of state-of-the-art methods to detect and recognize text instances from large-scale street view images, closing the gap between research benchmarks and real applications. During the competition period, a total of 41 teams participated in the two proposed tasks with 132 valid submissions, i.e., text detection and end-to-end text spotting. This paper includes dataset descriptions, task definitions, evaluation protocols and results summaries of the ICDAR 2019-LSVT challenge.
I. INTRODUCTION
ICDAR 2019-LSVT addresses the limited scale of fully annotated scene-text data by introducing a large street-view dataset with full and weak annotations. The competition evaluates text detection and end-to-end text spotting to connect benchmark research with real applications.
- Dataset and motivation: 30,000 training images have full annotations, while 400,000 training samples provide weak annotations for exploring methods that exploit partial labels.Weakly annotated images generally provide one transcription per image without text-location annotations.
- Dataset and motivation: 450,000 total images, including 20,000 testing images, make LSVT the largest scene-text dataset described in the competition.The images were captured in streets across complicated real-world scenarios.
- Dataset and motivation: At least 14 times larger than existing robust-reading benchmarks, LSVT is also presented as the first scene-text dataset with partial annotations for detection and recognition challenges.The dataset combines fully annotated data, large-scale weakly annotated training data, street-captured text, and a large testing benchmark.
III. DATASETS
LSVT contains 450,000 street-captured Chinese-text images gathered from different users and phones, combining fully annotated images with weak keyword transcriptions. Full annotations encode text geometry and transcription, while weak annotations omit locations.
- Image source and scale: 450,000 images with text were freely captured in Chinese streets, including storefronts and landmarks, using different users and mobile phones.Faces and license plates were algorithmically detected and blurred for privacy.
- Image source and scale: 50,000 images are fully annotated, split into 30,000 training images and 20,000 test images; the remaining 400,000 images are weakly annotated.The dataset is described as more than 14 times larger than existing robust-reading benchmarks.
- Annotation format: Full annotations provide manually labeled coordinates and transcriptions, using quadrilaterals for straight text and 8- or 12-vertex polygons for curved text.Horizontal, multi-oriented, and vertical instances use quadrilateral boxes, while curved instances use polygons arranged clockwise from the reading direction.
- Annotation format: Weakly annotated images provide only keyword transcriptions, while illegible regions are marked Do Not Care and excluded from final-score calculations.The keywords are referred to as weak annotations.
IV. TASKS
The competition evaluates large-scale street-view text reading through two tasks: locating text instances and performing end-to-end text spotting.
- Task definitions: The two tasks are text detection and end-to-end text spotting.They are introduced as common tasks for evaluating text-reading performance on the large-scale street-view benchmark.
A. TASK 1 - TEXT DETECTION
The text-detection task measures whether predicted boxes or polygons correctly locate text instances, using IoU-based matching and Precision, Recall, and H-mean metrics.
- Task definition: Text detection identifies text-instance locations using bounding boxes or polygons.The task follows ICDAR 2015 evaluation protocols.
- Evaluation protocol: IoU thresholds of 0.5 and 0.7 are used for evaluation, while H-mean at IoU greater than 0.5 is the final ranking metric.A detection is a true positive when its region has more than 0.5 IoU with the ground truth.
- Evaluation protocol: When multiple detections match one ground truth, only the detection with the highest IoU is retained and the others count as false positives.Detected or missed Do Not Care ground truths do not contribute to evaluation.
- Evaluation metrics: Precision, Recall, and H-mean are computed from true positives, false positives, and false negatives.TP, FP, and FN denote true positive, false positive, and false negative, respectively.
B. TASK 2 - END-TO-END TEXT SPOTTING
Task 2 evaluates end-to-end text spotting by requiring both text locations and recognized transcriptions, using exact-match and normalized edit-distance criteria. The protocol pairs predictions with ground truths by overlap and handles unmatched regions and annotation ambiguities explicitly.
- End-to-end text spotting requires submitting every text instance’s quadrangle or polygon location together with its recognized text.
- Evaluation reports Precision, Recall, H-mean, and Norm, where Norm is defined as 1 − N.E.D.Under exact matching, a true positive requires IoU higher than 0.5 and zero Levenshtein distance.
- Norm averages normalized Levenshtein distances over paired predicted and ground-truth regions, including unmatched regions paired with empty strings.
- String comparison is case-insensitive for English, equates traditional and simplified Chinese characters, and ignores spaces and symbols.
- Weakly annotated samples provide important keywords without locations, making them cheaper to collect and annotate than fully labeled samples.
V. SUBMISSIONS
The submissions section summarizes participation across the competition’s two tasks, reporting 132 valid submissions from 41 teams.
- 132 valid submissions came from 41 teams across the two competition tasks.
A. Top 3 submissions in Task 1
The leading Task 1 detection submissions predominantly build on Mask R-CNN-style detectors, adding multi-scale features, deformable convolutions, hard-example mining, or multi-scale testing.
- Tencent-DPPR used Mask R-CNN and PixelLink detectors with feature-pyramid proposal selection, multi-scale testing, and model ensembles.
- NJU ImagineLab extended Mask R-CNN with a ResNet-152 backbone, deformable ConvNets v2, PANet feature fusion, OHEM, and multi-scale testing.
- PMTD used a modified Pyramid Mask Text Detector based on Mask R-CNN and adopted multi-scale testing.
B. Top 3 submissions in Task 2
The leading Task 2 submissions use two-stage detection-recognition pipelines with CNN/RNN-style recognition, CTC or attention decoding, feature enhancements, and model ensembles.
- Tencent-DPPR combined two-stage detection with CNN features and Bi-GRU, attention-based RNN, and self-attention recognition models.Its ensemble training used synthetic data and several public scene-text datasets.
- HUST VLRGROUP used Mask TextSpotter with ResNet-50-FPN, deformable convolutions, Cascade R-CNN proposals, and CNN-RNN recognition with CTC loss.
VI. ANALYSIS
The competition analysis finds that instance-level segmentation and model enhancements perform strongly in detection, while end-to-end systems commonly use two-stage detection-recognition pipelines. Remaining errors include vertical-text grouping, difficult visual conditions, and the limited exploitation of detection-recognition feedback.
- Text detection: Most Task 1 participants use Mask R-CNN-style instance segmentation, enhanced backbones, deformable convolutions, and PANet for localization.These components target accurate localization, including long and curved text instances.
- Text detection: Tencent-DPPR achieves the best Task 1 H-mean, recall, and precision with enhanced backbones and model ensembles.
- Text detection: Vertical text lines are commonly misdetected or grouped as horizontal lines, producing detection errors.The analysis suggests semantic information may help decide whether to group or split text instances.
- End-to-end text spotting: Most Task 2 methods separate detection and recognition in a two-stage pipeline, using CNN and recurrent encoders with CTC-based prediction.Tencent-DPPR ranks first in precision, H-mean, and Norm through ensembles, while HUST leads recall using STN-based rectification.
- End-to-end text spotting: End-to-end recognition remains challenged by complex backgrounds, low luminance contrast, partial character occlusions, and varied text geometry.The analysis notes that detection and recognition are complementary, but the common pipeline does not fully exploit their relationship.
VII. CONCLUSIONS AND FUTURE DIRECTIONS
The paper summarizes the ICDAR 2019-LSVT challenge, its large-scale fully and weakly annotated dataset, and results from two tasks. Future leaderboard maintenance is intended to encourage continued submissions and help bridge research with industrial applications.
- Conclusions: ICDAR 2019-LSVT provides the largest ICDAR scene text dataset, with full and weak annotations supporting two challenge tasks.The challenge attracted 41 teams and produced 132 valid submissions.
- Conclusions: Results analysis summarizes the capabilities, difficulties, remaining challenges, and future research directions for state-of-the-art text reading systems.
- Future directions: Maintaining the competition leaderboard is intended to encourage more submissions and improvements while helping bridge research and industrial applications.