Source-linked AI summary
ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)
Chee-Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, Chee Seng Chan, Lianwen Jin
TL;DR
ArT addresses the scarcity and weak performance of scene text systems on curved and irregular text by providing a larger arbitrary-shaped-text dataset and benchmark. The challenge evaluates detection, recognition, and spotting, with leading scores of 82.65% for detection, 74.30% for recognition, and 52.45% for Task 3.1.
Problem
Existing benchmarks rarely represent curved or irregular text, and state-of-the-art detectors perform poorly on such data.
Method
The paper presents the ArT dataset and ICDAR2019 challenge, combining existing curved-text datasets with newly collected images and defining tasks, metrics, and participant evaluations.
Results
The challenge reports leading scores of 82.65% for Task 1, 74.30% for Task 2.1, 85.32% for Task 2.2, 52.45% for Task 3.1, and 44.91% for Task 3.2.
Takeaways & Limitations
ArT encourages academic and industrial research on arbitrary orientation and shape in text instances in the wild.
Abstract
from arXiv · showhide
This paper reports the ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT) that consists of three major challenges: i) scene text detection, ii) scene text recognition, and iii) scene text spotting. A total of 78 submissions from 46 unique teams/individuals were received for this competition. The top performing score of each challenge is as follows: i) T1 - 82.65%, ii) T2.1 - 74.3%, iii) T2.2 - 85.32%, iv) T3.1 - 53.86%, and v) T3.2 - 54.91%. Apart from the results, this paper also details the ArT dataset, tasks description, evaluation metrics and participants methods. The dataset, the evaluation kit as well as the results are publicly available at https://rrc.cvc.uab.es/?ch=14
I. INTRODUCTION
ArT addresses the underrepresentation and poor handling of curved and irregular text by creating a larger benchmark and organizing detection, recognition, and spotting tasks for arbitrary-shaped text.
- Existing scene text datasets are dominated by horizontal or rotated linear text, while curved and irregular arrangements are rarely represented.
- Recent studies report that state-of-the-art scene text detectors perform poorly on arbitrary-shaped text, motivating major design changes.
- The competition expands training and testing data by combining Total-Text and SCUT-CTW1500 images with newly collected images containing arbitrary-shaped, horizontal, and multi-oriented text.
- ArT comprises scene text detection, scene text recognition, and scene text spotting tasks, emphasizing robustness to arbitrary text shapes.
- The paper describes the related work, ArT dataset, task definitions, evaluation metrics, participant results, and conclusions.
II. RELATED WORK
Scene text benchmarks have grown in scale, difficulty, and annotation flexibility, while newer datasets specifically address the scarcity of curved text using polygonal ground truth.
- Scene text benchmarks have progressed from small rectangular datasets toward larger, more difficult datasets with flexible annotations.ICDAR2013 had 462 images, ICDAR2015 had 1,500, and IC17-MLT increased to 18,000 images.
- Curved text is common in real-world scenes but remains rare in existing benchmarks and is often loosely annotated with rectangular regions.
- Total-Text and SCUT-CTW1500 were introduced with emphasis on curved text and polygonal annotations.
- ArT combines Total-Text, SCUT-CTW1500, Baidu Curved Text Dataset, and newly collected images.
- ArT images come from cameras, online sources, image libraries, Google Open-Image, and Baidu Street View, covering indoor and outdoor scenes.
1) Type/source of images:
ArT uses existing real-world curved-text datasets together with a large newly collected image set to broaden coverage of arbitrary-shaped text.
- 4) Ground truth:: The source datasets consist of real-world scenes and are mostly well focused, while SCUT-CTW1500 includes Chinese-script annotations unlike Total-Text.
- 4) Ground truth:: Total-Text ground truth was refined to annotate all Chinese characters, and the datasets required annotation adjustments for Latin-script line levels.
- 4) Ground truth:: ArT combines 3,055 existing images from Total-Text and SCUT-CTW1500 with 7,111 newly added images.
- 4) Ground truth:: The dataset contains 10,166 images, split into 5,603 training images and 4,563 newly collected testing images.
4) Ground truth:
ArT uses tight polygon annotations instead of the axis-aligned boxes or quadrilaterals used by earlier RRC datasets, matching arbitrary-oriented and curved text.
- 4) Ground truth:: ArT uses polygon ground truth because fixed two- or four-vertex annotations are unsuitable for arbitrary-oriented Chinese and Latin text, especially curved instances.
- 4) Ground truth:: The competition was jointly organized by the University of Malaya, South China University of Technology, Baidu, and the Computer Vision Centre.
V. TASK 1: SCENE TEXT DETECTION
Task 1 evaluates scene text detection by requiring predicted text locations and confidence scores, using IoU-based H-mean ranking at a 0.5 threshold. Figure 1 summarizes the distribution of average scores across test images.
- Task 1 requires each submission to provide the spatial location and confidence score of every predicted text instance.
- IoU-based evaluation reports results at 0.5 and 0.7 thresholds, but official ranking uses H-Mean at 0.5.Confidence thresholds are iterated to find the best H-Mean score.
- Figure 1 plots a histogram of the average score achieved by all submissions for each test-set image.
C. Results and Discussion
Task 1 received 48 submissions and was led by Pil-Mask-RCNN, while score distributions and qualitative examples reveal both strong performance and recurring detection failures.
- Task 1: 82.65% was the winning H-mean score for Task 1, achieved by Pil-Mask-RCNN.The task received 48 submissions from 35 unique participants, and its average H-mean was 67.46%.
- Task 1: Most top-performing Task 1 methods used Mask R-CNN, and most participants produced polygon bounding regions.Polygon outputs align with the competition’s emphasis on tight detection boundaries.
- Task 1: Common Pil-Mask-RCNN failures were under-segmentation, grouping crowded text instances, and missing small text instances.
- Task 1: Most Task 1 test images had average H-mean scores between 0.8 and 0.9, while some challenging images scored 0 to 0.1.Figure 2 presents examples of images with low average H-mean scores.
- Task 1: Figure 3 contrasts successful and failed Pil-Mask-RCNN detections, marking true positives in green and false positives or false negatives in red.
B. Evaluation metrics
Recognition performance is evaluated separately for Latin and mixed Latin–Chinese scripts using task-specific scoring rules.
- Task 2.1: Task 2.1 uses case-insensitive word accuracy as its primary challenge metric.Standard recognition evaluation practices are also applied, including handling of symbols at text boundaries.
- Task 2.2: Task 2.2 uses 1-N.E.D., the Normalized Edit Distance metric, as its ranking metric.
- Recognition submissions are evaluated under Latin-only and mixed Latin-and-Chinese script categories.For Latin-script evaluation, non-Latin transcriptions are treated as “Do Not Care” regions.
C. Results and Discussion
Task 2 recognition results show distinct leaders for Latin-only and mixed Latin–Chinese recognition, with rectification and attention-based recognition recurring among strong methods.
- Task 2.1: 74.30% was the winning score for Task 2.1, achieved by PKU Team Zero using detection, spatial transformation, and attention RNN recognition modules.The spatial transformer straightens cropped text using polygon coordinates before recognition.
- Common pipeline: Task 2.1 winners rectified cropped patches before recognition and used attention mechanisms in their recurrent recognition modules.The reported pipelines first straighten text regions and remove background before word recognition.
- Qualitative results: Figure 4 presents successful and failed recognition examples from PKU Team Zero.The examples are qualitative illustrations of the method’s recognition behavior.
- Task 2.2: 85.32% was the winning score for Task 2.2, achieved by CRAFT (Preprocessing) + TPS-ResNet.Its pipeline combines CRAFT detection, TPS-based normalization, and BiLSTM recognition with attention.
VII. TASK 3: SCENE TEXT SPOTTING
Task 3 evaluates end-to-end text spotting by jointly locating and recognizing text instances, with outputs specified at word-level for Latin and line-level for Chinese script.
- Task definition: Task 3 requires end-to-end detection and recognition of every text instance in an input image.Outputs include each instance’s spatial location and predicted word, with Latin text represented at word-level and Chinese text at line-level.
- Qualitative examples: Figure 5 shows successful and failed recognition examples for CRAFT (Preprocessing) + TPS-ResNet.The figure caption identifies panels for successful and failure cases.
B. Evaluation metrics
Task 3 uses IoU-based detection matching followed by recognition evaluation, and its results show that end-to-end spotting remains especially challenging.
- Evaluation procedure: Task 3 matching pairs detections with ground truth when IoU exceeds 0.5, retaining only the highest-IoU detection when multiple matches occur.Other matched detections are counted as false positives, and Task 3.1 ignores Chinese text regions.
- Task 3 results: 52.45% was the Task 3.1 winning H-mean score, achieved by baseline 0.5 class 5435.The winning method used a segmentation-based detector, an attention-based recognizer, and 5,435 recognition classes.
- Task 3 results: 44.91% was the average 1-N.E.D score for Task 3.2, slightly higher than Task 3.1’s average H-mean score of 44.37%.The top three winners were the same across Task 3.1 and Task 3.2.
- Qualitative examples: Figure 6 contrasts successful and failed examples from baseline 0.5 class 5435.The caption identifies successful examples on the left and failures on the right.
- Failure analysis: Four vertical red regions were evaluated as false positives although the ground truth contained two left-to-right Chinese text regions.The paper suggests that language-specific semantic information could potentially address this example.
- Score distribution: Task 3 has the most distributed score histogram, with most images scoring between 0.4 and 0.6 on average.The paper identifies this distribution as reflecting the task’s challenging nature.
VIII. CONCLUSIONS
ArT attracted substantial participation to arbitrary-shaped scene text understanding, while exposing unresolved evaluation and application gaps. The results indicate that text spotting remains the most challenging task, and current IoU evaluation can overreward incomplete detections.
- ArT drew strong participation despite arbitrary-shaped text having been underrepresented in earlier scene text benchmarks.The challenge was motivated by the limited prior attention to curved text and aimed to engage both academic and industrial communities.
- Current scene text understanding still has a gap between research advances and application needs.
- Segmentation-based methods appeared to dominate arbitrary-shaped text detection, while recognition systems commonly rectified text patches before attention-based sequence recognition.
- The current IoU metric can reward detections that miss several characters with 100% recall, motivating consideration of more suitable metrics such as TIoU.
- Text spotting was the most challenging task, with the lowest winning H-mean score among the reported tasks.