PaperlayerView exact revision on arXiv

Source-linked AI summary

COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images

Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, Serge Belongie

arXiv:1601.07140v2cs.CV

TL;DR

Scene-text detection and recognition in unconstrained natural images lacks sufficiently broad evaluation data. The paper introduces COCO-Text from naturally occurring MS COCO scenes, annotates text across multiple attributes, and evaluates photo OCR systems. The dataset exposes good precision and transcription accuracy but substantially weaker detection recall, especially for illegible text, motivating further work.

  • Problem

    Scene-text evaluation typically relies on small datasets with mostly iconic text, limiting coverage of unconstrained natural-image text.

  • Method

    The paper constructs COCO-Text from MS COCO, adding text boxes, legibility, category, script, and transcriptions, then evaluates three photo OCR approaches.

  • Results

    Photo OCR methods show good precision and recognition accuracy, but detection recall is weak overall, particularly for illegible text.

  • Takeaways & Limitations

    COCO-Text provides a large, diverse natural-image benchmark that reveals shortcomings in current scene-text detection and recognition systems.

Abstract

from arXiv · show

This paper describes the COCO-Text dataset. In recent years large-scale datasets like SUN and Imagenet drove the advancement of scene understanding and object recognition. The goal of COCO-Text is to advance state-of-the-art in text detection and recognition in natural images. The dataset is based on the MS COCO dataset, which contains images of complex everyday scenes. The images were not collected with text in mind and thus contain a broad variety of text instances. To reflect the diversity of text in natural scenes, we annotate text with (a) location in terms of a bounding box, (b) fine-grained classification into machine printed text and handwritten text, (c) classification into legible and illegible text, (d) script of the text and (e) transcriptions of legible text. The dataset contains over 173k text annotations in over 63k images. We provide a statistical analysis of the accuracy of our annotations. In addition, we present an analysis of three leading state-of-the-art photo Optical Character Recognition (OCR) approaches on our dataset. While scene text detection and recognition enjoys strong advances in recent years, we identify significant shortcomings motivating future work.

1. Introduction

COCO-Text addresses the difficulty of detecting and recognizing diverse text in unconstrained natural images by introducing a large-scale dataset built from MS COCO.

  • 1. Introduction: Natural-scene text recognition remains challenging, despite its potential value for assistive technology, robot navigation, and geo-localization.Existing evaluations typically use datasets with mostly iconic text and at most hundreds of images.
  • 1. Introduction: COCO-Text uses MS COCO images collected without text in mind, providing diverse text in complex everyday scenes and contextual object annotations.The dataset combines rich text annotations with MS COCO object annotations for contextual reasoning.
  • 1. Introduction: The dataset annotates text locations, legibility, text category, script, and transcriptions for legible western-script text.Categories include machine printed and handwritten text, while legibility distinguishes readable from unreadable instances.
  • 1. Introduction: 63,686 images contain 173,589 labeled text regions, supporting evaluation of scene text detection and recognition methods.The paper also analyzes three leading photo OCR algorithms on the dataset.
  • 1. Introduction: Photo OCR methods achieve good precision and transcription accuracy, but detection recall is considerably degraded, especially for illegible text.None of the evaluated methods provides viable functionality for illegible text.

2. Related Work

COCO-Text extends earlier scene-text resources by pairing natural-context MS COCO images with broad text coverage, contextual object annotations, and a scalable annotation strategy.

  • 2. Related Work: COCO-Text is based on MS COCO, whose images were collected for natural object contexts rather than specifically for text.This design supplies rich object and caption annotations alongside text data.
  • 2. Related Work: Earlier scene-text datasets are smaller or more specialized, often using iconic, cropped, wearable-device, or synthetically generated word images.Examples include ICDAR 03, ICDAR 15, Street View Text, IIIT 5k-word, and Synth90k.
  • 2. Related Work: Scene-text recognition systems generally detect proposal regions first and then recognize words within those regions.Representative approaches combine text detectors, character classifiers, segmentation or region grouping, and language models.
  • 2. Related Work: Related work also combines textual and visual cues for fine-grained image classification and studies strategies for improving noisy human labeling.These lines of work connect scene text with downstream recognition and annotation quality.
  • 2. Related Work: COCO-Text adapts constrained-budget task allocation by increasing annotation redundancy mainly for supposedly difficult annotations.Unlike some prior frameworks, it uses initial detector outputs rather than detector access during annotation.

3. Text Annotation

The annotation pipeline combines OCR suggestions, targeted crowdsourcing, quality control, and staged text labeling to build comprehensive COCO-Text annotations efficiently.

  • 3. Text Annotation: The pipeline separates text detection, fine-grained classification, and transcription into atomic tasks supported by Mechanical Turk workers and MS COCO metadata.Three leading photo OCR systems also provide detections and transcriptions to reduce crowd workload.
  • 3. Text Annotation: Quality control uses tutorials, qualification examples, worker image limits, catch trials, and time penalties for low-quality answers.These mechanisms aim to balance annotation quality, worker variety, and cost.
  • 3. Text Region Detection: Text detection records bounding boxes, targeting one box per legible word and one box per continuous illegible text region.OCR detections are retained only when at least one human annotator agrees, contributing about 20% of text annotations.
  • 3. Text Region Detection: Human spotting targets text missed by OCR through repeated review, highlighted prior detections, scene complexity, and object categories associated with text.Images with unusually few initial detections or many detected regions receive additional attention.
  • 3. Text Region Detection: False-positive proposals are filtered with crop grids and majority votes from three workers.Workers inspect region crops and select proposals that do not contain text.
  • 3. Text Classification and Recognition: Classification labels legibility, script, and printed-versus-handwritten type using repeated selection and majority voting, while transcription uses OCR checks and human iterations.Workers can mark unreadable regions illegible during transcription, and three-stage transcription checking retains accepted results.
  • 3.4. Annotation Performance Analysis: 57% of all text regions were detected by crowd annotators, including 84% of legible and 39% of illegible text.The result illustrates the difficulty of detecting text in COCO images even for human annotators.
  • 3.4. Annotation Performance Analysis: Classification accuracy reached 93% for printed-versus-handwritten type, 87% for legibility, and 99% for script; 87.5% of transcriptions matched ground truth exactly.92.5% of transcriptions were within edit distance 1.

4. Dataset Statistics

COCO-Text differs from prior scene-text datasets through its natural-image origins, broader text distribution, contextual object annotations, and larger scale.

  • 50% of COCO-Text images contain no text, while images with text contain an average of 5.46 text instances.Across all images, the average is 2.73 text instances.
  • COCO-Text has a wider spatial text-occurrence distribution than related datasets because its images were not collected with text in mind.
  • 60.3% of text is legible and 39.7% illegible, while 86.4% is machine printed and 4.6% handwritten.The remaining 9% is borderline or from other categories.
  • COCO-Text contains contextual MS COCO object annotations, enabling analysis of relationships between scene text and objects.The paper reports that certain object categories are informative regarding text presence.
  • COCO-Text contains 63,686 images and 173,589 text annotations, making it more than 14 times larger than ICDAR 15.

5. Algorithmic Analysis

The paper evaluates three anonymized state-of-the-art photo OCR systems for detection, transcription, and end-to-end spotting on natural images, finding strong precision or recognition in some cases but weak detection overall.

  • Scene-text detection is evaluated using an Intersection-over-Union threshold of 50%, following the ICDAR robust-reading evaluation scheme.
  • Three state-of-the-art photo OCR algorithms are evaluated on detection, transcription, and end-to-end text spotting.The methods come from collaborators at Google, TextSpotter, and VGG and are anonymized in Table 2.
  • 83.78% and 89.73% detection precision are achieved by methods A and B, respectively.
  • 82.91% recognition accuracy is achieved by method A.
  • 34.01% is the best reported detection result for legible machine-printed text, and no method performs satisfactorily overall.
  • Detection results are even lower for legible handwritten text, and no method provides viable functionality for illegible text.The paper notes that current photo OCR algorithms are not intended to detect or transcribe illegible text.

6. Dataset Split

COCO-Text is divided into training and validation sets, with guidance on which annotations count for end-to-end text-spotting evaluation.

  • The training and validation sets contain 43,686 and 20,000 images, respectively.
  • End-to-end text-spotting results should consider only legible machine-printed and handwritten text.
  • Researchers are encouraged to train and tune parameters on the training set while minimizing evaluation-set runs.

7. Discussion

COCO-Text introduces a large-scale, diverse resource for natural-image text and uses photo OCR evaluation to expose important detection shortcomings that motivate future work.

  • COCO-Text was created from over 1,500 worker hours and includes text instances spanning several text types.
  • The dataset is presented as the first large-scale dataset for text in natural images and the first to annotate attributes such as legibility and text type.
  • Dataset statistics show broad text diversity and a wider spatial distribution than related datasets.
  • Photo OCR methods show satisfactory precision but significant shortcomings, especially in detection recall.
  • The discussion motivates algorithms capable of detecting wider varieties of text.
Loading 1601.07140v2…