Source-linked AI summary
ICDAR2017 Competition on Reading Chinese Text in the Wild (RCTW-17)
Baoguang Shi, Cong Yao, Minghui Liao, Mingkun Yang, Pei Xu, Linyan Cui, Serge Belongie, Shijian Lu, Xiang Bai
TL;DR
Chinese text reading remains less studied than English despite Chinese text’s distinct structural characteristics and practical value. RCTW-17 addresses this gap through a competition with a large annotated Chinese scene-text dataset and localization and end-to-end recognition tasks, whose analysis highlights localization quality and perspective distortion as recognition challenges.
Problem
Chinese text reading is less studied than English, despite Chinese’s larger character set, lack of word spaces, and multi-part characters, alongside its practical value.
Method
The paper organizes a Chinese text-reading competition using the 12,263-image CTW-12k dataset with line-level polygon and transcription annotations and two tasks: localization and end-to-end recognition.
Results
End-to-end recognition depends on text localization, while perspective distortions substantially harm recognition performance.
Takeaways & Limitations
RCTW-17 provides a dedicated dataset, evaluation setting, and analysis that shed light on Chinese text-reading challenges and reflect broad community interest.
Takeaways & Limitations
The authors plan to correct annotation mistakes and add images to CTW-12k, indicating that the dataset is not yet final.
Abstract
from arXiv · showhide
Chinese is the most widely used language in the world. Algorithms that read Chinese text in natural images facilitate applications of various kinds. Despite the large potential value, datasets and competitions in the past primarily focus on English, which bares very different characteristics than Chinese. This report introduces RCTW, a new competition that focuses on Chinese text reading. The competition features a large-scale dataset with 12,263 annotated images. Two tasks, namely text localization and end-to-end recognition, are set up. The competition took place from January 20 to May 31, 2017. 23 valid submissions were received from 19 teams. This report includes dataset description, task definitions, evaluation protocols, and results summaries and analysis. Through this competition, we call for more future research on the Chinese text reading problem. The official website for the competition is http://rctw.vlrlab.net
I. INTRODUCTION
RCTW-17 addresses the under-studied problem of reading Chinese text in natural images, whose linguistic and visual characteristics differ from English. It introduces a large annotated dataset and two competition tasks to stimulate further research.
- I. INTRODUCTION: Chinese text reading remains less studied despite Chinese’s widespread use and practical value.Chinese has a larger character set, lacks spaces between words, and often contains characters made of multiple non-connected parts.
- I. INTRODUCTION: 12,263 annotated images form a new Chinese scene-text database with text locations and transcriptions.
- I. INTRODUCTION: The competition evaluates text localization and end-to-end recognition.
- I. INTRODUCTION: 59 teams registered, 19 submitted results, and the report presents evaluation results and analysis.
- I. INTRODUCTION: The dataset and tasks aim to encourage future research and development on Chinese text reading.
II. DATASET AND ANNOTATIONS
CTW-12k contains 12,263 Chinese-text images with line-level polygon and transcription annotations, spanning natural and digital-born sources. The competition uses these data for localization and end-to-end recognition rather than cropped recognition.
- II. DATASET AND ANNOTATIONS: CTW-12k comprises 12,263 images, mostly phone-camera natural images plus digital-born screenshots, each containing at least one Chinese text line.
- II. DATASET AND ANNOTATIONS: Every text line is annotated with a four-point polygon and UTF-8 transcription, while illegible lines receive a difficult flag.
- II. DATASET AND ANNOTATIONS: Annotations are made at the text-line level because Chinese words are not separated by blank spaces.
- II. DATASET AND ANNOTATIONS: The dataset includes diversity in image sources, fonts, layouts, and Chinese and English text.
- II. DATASET AND ANNOTATIONS: The trainval set contains 8,034 images and the test set contains 4,229 images.
- II. DATASET AND ANNOTATIONS: The competition includes text localization and end-to-end recognition, omitting a separate cropped text recognition task.
A. Task 1 - Text Localization
Task 1 localizes text instances with scored four-point polygons. It ranks submissions primarily by polygon-based AP, with matching determined by an IoU threshold and one-to-one assignment.
- A. Task 1 - Text Localization: Participants localize text instances with four-point polygons and provide a confidence score for each polygon.
- A. Task 1 - Text Localization: AP is the primary metric because text is the only foreground category, while maximum F-measure is also reported for compatibility.
- A. Task 1 - Text Localization: Polygon IoU replaces rectangle IoU because text is localized by polygons.
- A. Task 1 - Text Localization: A detection is a true positive when polygon IoU exceeds 0.5 and its groundtruth is not matched elsewhere.
- A. Task 1 - Text Localization: AP is invariant to the precision-recall trade-off and provides precision-recall curves as byproducts.
B. Task 2 - End-to-End Recognition
Task 2 jointly evaluates text localization and recognition through matched detection-transcription pairs. Its edit-distance protocol penalizes both recognition errors and localization failures, with greater penalties for longer text.
- B. Task 2 - End-to-End Recognition: Participants submit detection results together with recognized UTF-8 text, making the task jointly evaluate localization and recognition.
- B. Task 2 - End-to-End Recognition: Detections are matched to the groundtruth polygon with maximum IoU when IoU exceeds 0.5; duplicate matches are reduced to the maximum-IoU detection.
- B. Task 2 - End-to-End Recognition: The score sums edit distances over matching pairs and divides by the number of test images.
- B. Task 2 - End-to-End Recognition: False positives and false negatives incur penalties equal to the length of the groundtruth text.
- B. Task 2 - End-to-End Recognition: The normalized edit distance is NED(s1, s2) = edit dist(s1, s2)/max(l1, l2), followed by averaging 1 − NED across matching pairs.
- B. Task 2 - End-to-End Recognition: Difficult text instances are excluded during recognition evaluation but retained for localization evaluation.
- B. Task 2 - End-to-End Recognition: The protocol gives longer text instances larger penalties, demanding stronger long-text detection and recognition performance.
IV. ORGANIZATION
The competition opened its website, released training/validation and test data on a staged schedule, and received submissions from 19 teams.
- The competition website opened on January 20, 2017, providing information, dataset links, registration, and submission access.
- Training and validation data were released on February 15, while the test dataset was released on April 15, two weeks before the deadline.
- 19 teams submitted results, with all entering Task 1 and four also entering Task 2.
V. SUBMISSIONS AND RESULTS
Task 1 results summarized the top-10 submissions using AP and maximum F-measure, with rankings based on AP and similar rankings from F-measure.
- Task 1’s top-10 submissions were ranked by Average Precision, with Maximum F-measure also reported.
- Ranking by Maximum F-measure produced results similar to ranking by Average Precision.
- Foo&Bar used Faster R-CNN with a quadrangular regression layer, ResNet-101, and ImageNet pretraining.
- NLPR PAL used a Deep Direct Regression Network for boxes or long-line segments, grouped into text lines, and added 7,000 self-selected training images.
- gmh used a CNN-based method.
B. Top 3 submissions for Task 2
The Task 2 submissions used sequence-recognition and detection systems with synthetic or augmented training data, while the reported tables summarize task-specific rankings and metrics.
- Top 3 submissions for Task 2: NLPR PAL used sliding convolutional character models trained end to end on text-line transcripts, with normalized outputs decoded by refined beam search.
- Top 3 submissions for Task 2: Table II summarizes Task 2 submissions using Average Edit Distance and a normalized measure.
- Top 3 submissions for Task 2: NLPR PAL’s recognition system was purely trained with synthetic data covering 7,356 Chinese-character and symbol classes.
- Top 3 submissions for Task 2: SCUT DLVC combined two-stage quadrilateral detection with multiscale CNN, bidirectional LSTM, CTC recognition, perspective transformation, and synthesized extra data.
- Top 3 submissions for Task 2: CCFLAB used Faster R-CNN with ResNet-101, fused top-down feature maps, ROI pooling, and only the provided dataset during training.
C. Baseline submissions
The organizers provided baselines for both tasks, using SegLink-based detection and a modified CRNN recognition pipeline supported by synthetic pretraining.
- Baseline methods were submitted for Task 1 and Task 2, with results reported in Tables I and II.
- The Task 1 baseline used SegLink to predict multiscale segments and links, which were combined into text lines.
- The Task 2 baseline used a modified CRNN with convolutional features, bidirectional LSTM context modeling, and CTC transcription without prior character-level annotation.
- Detected text lines were cropped, classified as horizontal or vertical by length-width ratio, and processed by separate models pretrained on synthetic data with a Chinese lexicon.
VI. ANALYSIS
The analysis identifies recurring localization and recognition failures, including long-text detection, redundant detections, perspective distortion, and structurally similar Chinese characters.
- Localization: Digital-born images generally achieve better detection performance than natural images.The authors suggest cleaner backgrounds and simpler fonts as likely reasons.
- Localization: Long text lines are often incompletely detected or split into multiple pieces.The authors associate this difficulty with producing accurate boxes having large aspect ratios using popular object detection frameworks.
- Localization: Redundant detections are difficult to suppress when they are much smaller than whole text lines.The authors report that standard non-maximum suppression struggles in these cases.
- Recognition: End-to-end recognition depends on text localization, and worse localization produces worse recognition.Perspective distortions also harm recognition performance badly.
- Recognition: Characters with similar structures are commonly confused because their subtle differences are difficult to distinguish.Figure 5 illustrates recognition failures of this kind.
VII. CONCLUSION
The report presents RCTW-17 as a Chinese text-reading benchmark built around a new dataset, tasks, and evaluation protocols. It analyzes competition results and plans continued evaluation and dataset improvement.
- VII. CONCLUSION: RCTW-17 introduced a new Chinese scene-text dataset, two tasks, and evaluation protocols designed for Chinese text reading.The competition addressed text localization and end-to-end recognition.
- VII. CONCLUSION: The competition attracted broad community interest, and result analysis highlighted challenges and difficulties in Chinese text reading.The report states that it received a good number of registrations and submissions.
- VII. CONCLUSION: The organizers plan a continuous challenge with online submissions, evaluation, comparison, and ongoing CTW-12k annotation improvements.Planned annotation work includes correcting mistakes and adding new images.