PaperlayerView exact revision on arXiv

Source-linked AI summary

Scene Text Detection via Holistic, Multi-Channel Prediction

Cong Yao, Xiang Bai, Nong Sang, Xinyu Zhou, Shuchang Zhou, Zhimin Cao

arXiv:1606.09002v2cs.CV

TL;DR

Scene text detection must cope with highly variable text and clutter, while local detectors may miss wide-scope contextual cues. The paper uses holistic semantic segmentation with one FCN to predict text regions, characters, and their relationships, and reports superior benchmark performance plus a first COCO-Text baseline. Its limitations include sensitivity to severe blur and highlights, a roughly 56M model, and slow CPU inference.

  • Problem

    Local-region scene text detectors can exclude wide-scope context, while many methods overlook non-horizontal text despite its importance in natural scenes.

  • Method

    A single FCN performs holistic, image-level pixel-wise prediction of text regions, individual characters, and linking orientations as semantic-segmentation maps.

  • Results

    The method substantially outperforms prior state-of-the-art approaches on standard benchmarks and reports the first quantitative COCO-Text result; on ICDAR 2013 it achieves recall 0.8022.

  • Takeaways & Limitations

    Joint prediction of text properties enables detection of horizontal, multi-oriented, and curved text in real-world natural images.

  • Takeaways & Limitations

    The method is sensitive to severe blur and highlights, has a model size of about 56M, and takes more than 14s on CPU for 640x480 images.

Abstract

from arXiv · show

Recently, scene text detection has become an active research topic in computer vision and document analysis, because of its great importance and significant challenge. However, vast majority of the existing methods detect text within local regions, typically through extracting character, word or line level candidates followed by candidate aggregation and false positive elimination, which potentially exclude the effect of wide-scope and long-range contextual cues in the scene. To take full advantage of the rich information available in the whole natural image, we propose to localize text in a holistic manner, by casting scene text detection as a semantic segmentation problem. The proposed algorithm directly runs on full images and produces global, pixel-wise prediction maps, in which detections are subsequently formed. To better make use of the properties of text, three types of information regarding text region, individual characters and their relationship are estimated, with a single Fully Convolutional Network (FCN) model. With such predictions of text properties, the proposed algorithm can simultaneously handle horizontal, multi-oriented and curved text in real-world natural images. The experiments on standard benchmarks, including ICDAR 2013, ICDAR 2015 and MSRA-TD500, demonstrate that the proposed algorithm substantially outperforms previous state-of-the-art approaches. Moreover, we report the first baseline result on the recently-released, large-scale dataset COCO-Text.

I. INTRODUCTION

The paper recasts scene text detection as holistic semantic segmentation to exploit whole-image context and jointly predict text properties for separating and detecting varied text instances.

  • Scene text detection is challenging in natural images because text varies in appearance, layout, font, language, and style amid clutter, blur, occlusion, and uneven illumination.
  • Local-region methods have limited access to wider context, hindering detection of weak text and suppression of difficult false positives.
  • The proposed method performs holistic, per-pixel semantic segmentation on full images rather than local candidate extraction.
  • Three prediction types—text regions, individual characters, and adjacent-character relationships—help separate nearby instances and support multi-oriented and curved text detection.
  • The FCN-based strategy uses multi-scale learning and prediction to address text variability and hard false alarms in natural scenes.
  • Experiments on ICDAR 2013, ICDAR 2015, MSRA-TD500, and COCO-Text report performance exceeding prior state-of-the-art methods, including a first quantitative result on COCO-Text.

II. RELATED WORK

Prior scene text detectors largely use local components or windows and often focus on horizontal text. This work instead builds on holistic FCN/HED prediction to handle broader text variation and orientation.

  • Earlier methods commonly detect character candidates using edge or extremal-region extraction, or search for strokes with multi-scale sliding windows.
  • Most previous approaches focus on horizontal or near-horizontal text, limiting applicability when text appears in other orientations.
  • Deep learning methods improve performance, but the proposed approach differs by producing global, pixel-wise maps in a holistic fashion.
  • HED is adopted as the base model because holistic image-to-image prediction can use wide-scope context, while multi-scale and multi-orientation handling fits scene text.
  • The paper represents annotations with polygons and maps for text regions, characters, and linking orientations.

III. METHODOLOGY

The method uses a single FCN to make holistic, image-level predictions for text regions, characters, and character-linking orientations. Detections are then formed from these maps through segmentation, aggregation, and partitioning.

  • III. METHODOLOGY: The FCN estimates text regions, individual characters, and linking orientations as three image-level, pixel-wise prediction maps.The linking-orientation map represents relationships between adjacent characters.
  • III. METHODOLOGY: The original image is fed into the trained model, which produces text-region, character, and linking-orientation maps before detection formation.Detection formation performs segmentation, aggregation, and partition on the three maps.
  • III. METHODOLOGY: Ground-truth annotations are converted into three label maps compatible with the FCN/HED framework.Text-region and character maps are binary, while character masks are shrunk to half size to prevent predicted characters from sticking together.
  • III. METHODOLOGY: The linking-orientation label map assigns foreground pixels normalized orientation values in the range [0, 1].Orientations are first considered in [−π/2, π/2] and then shifted and normalized.
  • III. METHODOLOGY: The prediction model is trained with training images and their corresponding ground-truth maps, following the general HED training procedure.Differences from HED are described separately in the paper.

3) Prediction Map Generation:

The system generates dense maps for text regions, characters, and linking orientations, then converts character candidates into graph structures for text-line grouping. Graph partition uses character similarities and a straightness criterion, with a threshold extending the procedure to curved text.

  • 3) Prediction Map Generation:: The trained model generates text-region, character, and linking-orientation maps from the original image, with orientations valid only inside foreground text regions.Background orientations are omitted when the text-region probability is below 0.5.
  • 3) Prediction Map Generation:: Text-region and character candidates are obtained by adaptively thresholding their prediction maps.Because character scales are halved during training, estimated candidate radii are multiplied by 2.
  • 4) Detection Formation:: Characters within the same text region are grouped into cliques, triangulated with Delaunay triangulation, and represented as graphs whose edges model pairwise similarities.Delaunay triangulation eliminates unnecessary linkings between distant characters.
  • 4) Detection Formation:: Graph edge weights are based on pairwise character similarity, while non-triangulation linkings receive zero weight.The similarity combines spatial and orientation similarity.
  • 4) Detection Formation:: Orientation similarity compares the line between character centers with the average linking orientation between them, rewarding agreement and penalizing violations.The included angle is treated as an acute angle.
  • 4) Detection Formation:: A maximum spanning tree is partitioned by selecting and eliminating edges, producing groups corresponding to individual text lines.Removing one edge yields two text lines, while removing two yields three.
  • 4) Detection Formation:: The optimal text-line segmentation maximizes a straightness measure based on the two largest covariance-matrix eigenvalues for each cluster.The cluster count K represents the number of inferred text lines.
  • 4) Detection Formation:: A threshold τ prevents high-weight edges from being selected or eliminated, allowing the segmentation strategy to handle both linear and curved text lines.The threshold addresses the failure of the linearity assumption for curved text.

5) Post-Processing:

For datasets with word-level annotations, the method applies word partitioning during post-processing.

  • 5) Post-Processing:: Word partitioning is used for ICDAR 2013 and ICDAR 2015 because text instances in those datasets are labeled at the word level.The adopted method is described as simple and effective.

C. Architecture

The architecture adapts HED’s pretrained VGG-16-based network for holistic scene text detection by predicting text regions, characters, and linking orientations through multiple output channels. Training uses fused outputs and a weighted combination of losses for these three targets.

  • Architecture: The network modifies HED’s pretrained five-stage VGG-16 architecture for scene text detection.Each stage receives side-output layers connected to the relevant prediction channels.
  • Training targets: The training targets comprise binary text-region and character maps plus a linking-orientation map defined for foreground pixels.The model treats linking orientation as undefined in background regions.
  • Architecture: Each stage predicts text regions, individual characters, and linking orientations between adjacent characters.These three channels provide complementary text properties for separating and grouping scene text.
  • Training objective: The fused loss is a weighted sum of region, character, and orientation losses, with weights λ1, λ2, and λ3 summing to 1.The region loss uses class balancing, while the character loss follows the analogous formulation.
  • Training objective: The orientation loss is near zero for small angular errors, peaks near ±π/2, and ignores orientation errors at background pixels.Its double-peak shape reflects the orientation definition used by the method.
  • Testing: During testing, the mean-subtracted image is fed to the trained model, which outputs fused maps for regions, characters, and linking orientations.The model treats each image independently, and linking orientations are valid only within foreground regions.

2) Testing Phase:

The method differs from HED by using only fused-layer responses at test time, because side-output responses were found to add noise and unimportant details.

  • Testing Phase: Only fusion-layer responses are used instead of averaging side-output and fusion-layer responses as in HED.The authors report that side-output responses can introduce harmful noise and unimportant details into scene text prediction maps.

IV. EXPERIMENTS AND DISCUSSIONS

The experiments evaluate the method on three standard scene-text benchmarks and COCO-Text, covering horizontal, incidental, multi-oriented, multilingual, and large-scale natural-image settings. COCO-Text provides the first reported quantitative baseline from this work on a benchmark with 63,686 images and substantial variability.

  • Evaluation setup: The implementation was evaluated on three standard benchmarks and compared with other scene text detection methods.Experiments ran on a single Tesla K40m GPU using a Caffe-based implementation.
  • Datasets: ICDAR 2013 contains 229 training and 233 testing natural images with horizontally placed English text.This dataset represents a relatively constrained orientation setting.
  • Datasets: ICDAR 2015 contains 1,500 Google Glass images with text captured without prior user preference or intention, making instances usually skewed.The dataset evaluates detection under incidental image capture conditions.
  • Datasets: MSRA-TD500 contains 500 natural scene images with varying text directions and both Chinese and English text.Its 300-image training and 200-image test split reflects variability in text and background complexity.
  • Datasets: COCO-Text annotates 173,589 text instances across 63,686 images, with 43,686 training images and 20,000 validation/test images.The paper reports the first quantitative performance on this large-scale benchmark.

B. Implementation Details

The proposed model uses an FCN-based implementation and is evaluated across benchmark datasets, with examples spanning diverse text conditions. On ICDAR 2013, it achieves the highest recall among compared methods.

  • Implementation Details: The model is built by modifying HED and uses ImageNet-pretrained VGG-16 parameters.The learning rate is set to 1e-8, with λ1 = λ2 = λ3 = 1 and τ = 0.8 used in all experiments.
  • Implementation Details: Training combines 1,529 images from ICDAR 2013, ICDAR 2015, and MSRA-TD500, augmented by rotations at 10-degree intervals.Images are rescaled to a maximum dimension of 960 pixels before augmentation.
  • Qualitative Results: The method handles text with varied orientations, languages, fonts, colors, and scales across diverse benchmark scenarios.Examples also show some tolerance to non-uniform illumination, blur, local distractors, and connected strokes.
  • Quantitative Results: 0.8022 recall is the highest reported among methods evaluated on ICDAR 2013, while the proposed F-measure is slightly better than prior state-of-the-art methods.The improvement is less obvious than on ICDAR 2015 and MSRA-TD500 because ICDAR 2013 is more saturated and dominated by horizontal text.

3) Quantitative Results on ICDAR 2015:

On ICDAR 2015, the method obtains the highest recall and second-highest precision among compared methods. The supplied results also report COCO-Text performance and show stronger behavior on legible machine-printed text than on illegible handwritten text.

  • ICDAR 2015: 0.5869 recall is the highest and 0.7226 precision is the second highest among methods evaluated on ICDAR 2015.
  • COCO-Text: On COCO-Text, the proposed algorithm achieves 0.4323 precision, 0.271 recall, and 0.3331 F-measure.The reported baseline methods cannot be directly compared because they supported annotation and were evaluated on a different dataset split.
  • COCO-Text Sub-categories: Performance is much better on legible, machine-printed COCO-Text than on illegible, handwritten text.
  • Efficiency: The method takes about 0.42 seconds to produce prediction maps and 0.2 seconds for subsequent CPU processing on 640x480 images.The timing is reported for a K40m GPU.
  • Generalization: The model generalizes to Arabic, Hebrew, Korean, and Russian scripts despite training only on English and Chinese examples.
  • Generalization: The trained model also handles curved text even though curved instances are rarely represented in the training data.

E. Generalization Ability of Proposed Algorithm

The method generalizes beyond its training distribution, handling unseen scripts and curved text. However, it remains sensitive to severe blur and highlights, and its size and CPU speed limit deployment on low-end devices.

  • Generalization: The model generalizes well to scripts beyond its English and Chinese training examples, including Arabic, Hebrew, Korean, and Russian.
  • Generalization: The method handles curved text despite curved examples being rarely seen during training.
  • Limitations: The method may fail under severe blur and highlight conditions.The authors suggest that additional examples and augmentation could alleviate these issues.
  • Limitations: A model size of about 56M parameters and CPU processing above 14 seconds for 640x480 images limit suitability for low-end PCs or mobile devices.
  • Future Directions: Future work targets better architectures, richer character-shape labels, and acceleration techniques to improve task suitability and efficiency.
Loading 1606.09002v2…