PaperlayerView exact revision on arXiv

Source-linked AI summary

ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019

Nibal Nayef, Yash Patel, Michal Busta, Pinaki Nath Chowdhury, Dimosthenis Karatzas, Wafa Khlif, Jiri Matas, Umapada Pal, Jean-Christophe Burie, Cheng-lin Liu, Jean-Marc Ogier

arXiv:1907.00945v1cs.CV

TL;DR

Multilingual scene-text understanding needs benchmarks spanning languages and scripts, while existing resources have focused mainly on English or limited multilingual coverage. RRC-MLT-2019 extends the prior challenge with four tasks, expanded real and synthetic datasets, and an end-to-end baseline, and reports 60 submissions across the challenge.

  • Problem

    The challenge addresses the need to benchmark multilingual scene-text detection and recognition across multiple languages and scripts.

  • Method

    RRC-MLT-2019 organizes four tasks, provides real and synthetic multilingual datasets, and introduces an end-to-end baseline recognition method.

  • Results

    The competition received 60 submissions across the four tasks, indicating participation from the research and industrial communities.

  • Takeaways & Limitations

    The challenge establishes evaluation protocols and standard benchmarks intended to promote future research in multilingual scene-text understanding.

Abstract

from arXiv · show

With the growing cosmopolitan culture of modern cities, the need of robust Multi-Lingual scene Text (MLT) detection and recognition systems has never been more immense. With the goal to systematically benchmark and push the state-of-the-art forward, the proposed competition builds on top of the RRC-MLT-2017 with an additional end-to-end task, an additional language in the real images dataset, a large scale multi-lingual synthetic dataset to assist the training, and a baseline End-to-End recognition method. The real dataset consists of 20,000 images containing text from 10 languages. The challenge has 4 tasks covering various aspects of multi-lingual scene text: (a) text detection, (b) cropped word script classification, (c) joint text detection and script classification and (d) end-to-end detection and recognition. In total, the competition received 60 submissions from the research and industrial communities. This paper presents the dataset, the tasks and the findings of the presented RRC-MLT-2019 challenge.

I. INTRODUCTION AND RELATED WORK

RRC-MLT-2019 extends the 2017 multilingual scene-text challenge with a new end-to-end task, expanded data, and a baseline method. It benchmarks four related capabilities and attracted broad participation.

  • Motivation: Scene-text reading supports applications including visual assistance, map and geographic-information services, autonomous driving, indoor navigation, and visual search.These applications motivate robust scene-text detection and recognition in natural images.
  • Challenge extensions: The challenge extends RRC-MLT-2017 with end-to-end multilingual text detection and recognition, an additional language, synthetic training data, and a baseline method.The 2019 edition also re-opened the three earlier tasks on the new real dataset.
  • Dataset and benchmark: 20,000 natural scene images contain text from Arabic, Bangla, Chinese, Devanagari, English, French, German, Italian, Japanese, and Korean.The dataset establishes a multilingual benchmark spanning ten languages.
  • Paper scope: The paper describes the challenge organization, datasets, task protocols, participant methods, and results, with detailed descriptions limited to winning methods.Participant methods are otherwise listed by name in linked result tables.
  • Challenge organization: The four tasks cover text detection, cropped-word script classification, joint detection and script classification, and end-to-end text recognition.The first three tasks were re-opened from MLT-2017, while end-to-end recognition was newly introduced.
  • Participation: The competition received 60 submissions distributed across the four tasks: 25 for Task-1, 15 for Task-2, and 10 each for Tasks 3 and 4.Some participants submitted to multiple tasks or submitted multiple similar methods.

III. THE “RRC-MLT-2019” DATASETS

The MLT-2019 datasets combine a 20,000-image real collection with a synthetic training set, covering ten languages and multiple script classes. The real images reflect varied natural-scene text and include defined splits and evaluation exclusions.

  • Dataset overview: The challenge provides 20,000 real natural-scene images with embedded text in 10 languages and a synthetic dataset intended as assistive training data for Task-4.The synthetic dataset matches the scripts represented in the real dataset.
  • Real dataset: The real images include street signs, advertisements, shop names, passing vehicles, and text in user photographs collected with mobile cameras or from the Internet.Most images contain intentional focused text, while very small, blurry, or occluded unintentional text is ignored during evaluation.
  • Language coverage: Each language is represented in at least 2,000 images, although most images contain text from more than one language.The ten languages are Arabic, Bangla, Chinese, Devanagari, English, French, German, Italian, Japanese, and Korean.
  • Script classes: The languages map to seven principal scripts, supplemented by Symbols and Mixed classes for standalone symbols and words combining multiple scripts.The principal script classes are Arabic, Bangla, Chinese, Hindi, Japanese, Korean, and Latin.

3) Ground Truth (GT):

Ground truth is provided at word level with geometric, script, and transcription annotations, while synthetic data augments training through multilingual text rendering over natural backgrounds. The synthesis pipeline was modified to preserve ligatures in several scripts.

  • Ground-truth annotation: Each ground-truth word has a four-corner bounding box, a script class, and a Unicode transcription.Chinese and Japanese text is labeled at line level rather than word level.
  • Ground-truth annotation: Unreadable text regions caused by low resolution or other distortions are marked “don’t care” and excluded from evaluation.This exclusion applies to regions that annotators cannot read reliably.
  • Synthetic data motivation: The synthetic dataset complements real data because deep-learning scene-text systems require substantial labeled training data.It adapts an existing synthesis framework to multiple languages.
  • Synthetic data generation: The synthesis framework overlays text on natural background images while accounting for 3D scene geometry.The original framework uses prospective text regions, depth estimation, planar facets, and geometric alignment for rendering.
  • Multilingual rendering: The pipeline was changed because character-by-character rendering breaks ligatures in Arabic, Bangla, and Devanagari words.The authors made appropriate modifications to handle this multilingual rendering issue.
  • Synthetic dataset: The synthetic dataset contains 277,000 images with word- and character-level bounding boxes, transcriptions, and language classes.It uses the same script classes as the real dataset and includes thousands of images for each language.

IV. TASK-1: MULTI-LINGUAL TEXT DETECTION

Task-1 evaluates word-level localization of multilingual scene text using quadrilateral detections, filtered for “don’t care” regions and matched to ground truth with an overlap threshold.

  • Task description: Task-1 requires participants to localize each detected text word with a four-corner bounding box in 10,000 test images.The training set contains 10,000 images with ground-truth word boxes, and the test set contains another 10,000 images.
  • Evaluation protocol: A detection counts as correct when its bounding box overlaps the ground-truth box by more than 50% intersection over union.Precision, recall, and Hmean are computed from image-level matches pooled across the complete test set.
  • Evaluation protocol: Detected boxes are first compared with “don’t care” regions and discarded when the specified overlap condition is satisfied.This filtering step removes unreadable or otherwise ignored regions before matching detections to ground truth.
  • Evaluation caveat: The evaluation notes minor issues when ground-truth regions overlap “don’t care” regions, but reports few cases and no impact on global method evaluation.The paper identifies this as a possible area for improving the evaluation methods.
  • Evaluation protocol: Filtered detections are matched to ground-truth quadrilaterals, with each detection and ground-truth element allowed to participate in at most one match.The resulting matches determine the cumulative precision, recall, and f-measure.

C. Participant Methods and Results for Task-1

Task-1 participant systems are ranked by Hmean, with most methods relying on R-CNN variants or other established deep networks for multilingual text detection.

  • Results: Task-1 participant rankings are summarized according to Hmean, with each method linked to its detailed online description and results.The paper reports participant results for the multi-lingual text detection task.
  • Participant methods: Most participant methods, including the winners, use R-CNN variants such as masked, cascaded, or refined architectures.The paper presents this as evidence that R-CNN can be improved for highly accurate detection.
  • Participant methods: Other systems use established text-detection and recognition components including ResNet, EAST, RRPN, and FPN.These approaches include one or more previously used deep networks.

1) Winner Methods of Task-1:

Two methods share first place in Task-1: Tencent-DPPR Team and Multi-stage Text Detector, with the reported Hmean difference judged insignificant given possible ground-truth errors.

  • Winner methods: Two methods are ranked first, and their Hmean difference is not significant given possible errors in the ground truth.The tied winners are Tencent-DPPR Team and Multi-stage Text Detector.
  • Tencent-DPPR Team: Tencent-DPPR Team uses a Mask R-CNN detector for multi-oriented text, trained with RRC-MLT-2019 and MSRA-TD500 data.It applies multi-scale training and combines two backbones with different multi-scale testing approaches for ensemble results.
  • Multi-stage Text Detector: Multi-stage Text Detector modifies Mask R-CNN with a rotated proposal module and then applies a refinement stage.The two-stage design targets multi-oriented scene text detection and final-result refinement.

V. TASK-2: CROPPED WORD SCRIPT IDENTIFICATION

Task-2 evaluates script identification for cropped word images from multilingual scene text, using classification accuracy to rank participants. The dataset covers 10 languages and eight valid script IDs.

  • Task description: Task-2 asks participants to identify one script ID for each cropped word image.The images are extracted from full scene images using ground-truth word boxes.
  • Dataset: 89,177 training images and 102,462 test images comprise the cropped-word datasets.
  • Dataset: The dataset contains text in 10 languages, with some languages sharing scripts, and includes occasional punctuation and mathematical symbols.
  • Task description: Arabic, Bangla, Chinese, Hindi, Japanese, Korean, Latin, and Symbols are the valid script IDs.
  • Evaluation and results: Classification accuracy determines evaluation and participant ranking, with Table II summarizing the results.
  • Participant methods: Participant methods mainly use deep recognition networks with additions such as multiscale processing, attention, voting, and script statistics.

1) Winner Method of Task-2:

The Task-2 winner, Tencent-DPPR Team, uses an ensemble of recognition models to identify text lines and character-level language types. Task-3 reuses this script-classification capability alongside text localization.

  • Winner method: Tencent-DPPR Team’s Task-2 method first recognizes text lines and their character-level language types using an ensemble of recognition models.The models are based on Seq2Seq with CTC and other components, although the supplied passage is truncated before the full description.
  • Task-3 connection: Task-3 requires localizing every word in a scene and jointly identifying the script ID of each detected word.Its training and test sets each contain 10,000 images, using the same scene images as Task-1.
  • Task-3 results: Table III summarizes participant rankings for joint text detection and script identification.
  • Task-3 evaluation: Task-3 evaluation counts a detection as correct only when both the bounding box and script ID are correct.The task otherwise follows Task-1’s ranking and evaluation protocol.

C. Participant Methods and Results for Task-3

Task-3 results are reported for joint text detection and script identification, within a broader shift toward end-to-end approaches. Tencent-DPPR Team wins with a cascade of its Task-1 detector and Task-2 classifier.

  • Results: Table III reports Task-3 participant rankings according to Hmean for joint text detection and script identification.
  • Participant methods: Most Task-1 participants also entered Tasks 3 and 4, reflecting a move toward end-to-end approaches built on deep learning methods.
  • Winner method: Tencent-DPPR Team wins Task-3 with a cascade of its Task-1 and Task-2 methods.The supplied passage describes Task-3 as a joint task combining detection and script classification.
  • Task-4 context: The newly introduced Task-4 addresses unified multilingual end-to-end scene text detection and recognition.Its objective is to localize bounding boxes and produce corresponding transcriptions.
  • Task-4 context: Table IV reports Task-4 results for end-to-end text detection and recognition.
  • Task-4 context: Task-4 provides synthetic data alongside the real dataset because the real training data is unbalanced across languages.Training and test sets each contain 10,000 images.

B. Evaluation Protocol for Task-4

Task-4 evaluates end-to-end detection and recognition by requiring both correct localization and exact transcription matching. The challenge also provides an unranked baseline and excludes unseen test characters from evaluation.

  • Evaluation protocol: Task-4 counts a detection as correct only when its text box is localized correctly and its transcription is recognized correctly.
  • Evaluation protocol: Task-4 uses Task-1’s ranking and evaluation protocol except for the definition of a correct detection.
  • Evaluation boundary: Test words containing characters absent from training are marked “don’t care” and excluded from both detection and recognition evaluation.Such words do not affect whether a method detects or recognizes them correctly.
  • Results reporting: The Task-4 ranking is summarized in Table IV, which also provides additional online metrics including edit-distance recognition accuracy.
  • Baseline: E2E-MLT is provided as a baseline with available source code and is not ranked because it was submitted by the organizers.
  • Participant methods: Participant systems generally combine detection and recognition deep networks with attention-based decoders, MORAN-v2, CRNN variants, and convolutional transformers.

1) Winner Method of Task-4:

Tencent-DPPR Team & USTB-PRIR won all four MLT-2019 tasks, while the challenge expanded the benchmark with new data and an end-to-end task. The report also identifies directions for broader language coverage and more robust evaluation.

  • Tencent-DPPR Team & USTB-PRIR won all four MLT-2019 tasks, sharing Task 1’s rank and collaborating on Task 4.
  • 60 submissions were distributed across the four tasks, indicating substantially increased community interest since the 2017 MLT edition.
  • The 2019 challenge extended MLT-2017 with a new language, an end-to-end recognition task, and a matching synthetic training dataset.
  • Future iterations should increase language coverage and improve evaluation for unfocused text and “don’t care” words in joint or end-to-end tasks.
Loading 1907.00945v1…