PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Oriented Scene Text Detection via Corner Localization and Region Segmentation

Pengyuan Lyu, Cong Yao, Wenhao Wu, Shuicheng Yan, Xiang Bai

arXiv:1802.08948v2cs.CV

TL;DR

Scene text detection must handle arbitrary orientations, large aspect ratios, and text-region ambiguity, while segmentation methods often require complex post-processing. The paper combines corner-point localization with position-sensitive segmentation to generate and score candidate boxes. It reports better or competitive accuracy and efficiency, including 84.3% F-measure on ICDAR2015 and 81.5% on MSRA-TD500, while noting failures for extremely close or curved text.

  • Problem

    Scene text detectors struggle with arbitrary orientation, large aspect-ratio variation, ambiguous boundaries, and complex post-processing for segmentation-based outputs.

  • Method

    The method detects text-box corner points, predicts relative-position segmentation maps, groups corners into candidate boxes, scores them with segmentation, and applies NMS.

  • Results

    The method achieves better or competitive accuracy and efficiency across public benchmarks, including 84.3% F-measure on ICDAR2015, 81.5% on MSRA-TD500, and processing above 10.4 512x512 images per second.

  • Takeaways & Limitations

    Corner-based localization naturally handles arbitrary-oriented and long text while position-sensitive segmentation supports text instances ranging from characters to lines.

  • Takeaways & Limitations

    The method may merge extremely close text instances and performs poorly on curved text because position-sensitive segmentation can fail and curved samples are scarce in training.

Abstract

from arXiv · show

Previous deep learning based state-of-the-art scene text detection methods can be roughly classified into two categories. The first category treats scene text as a type of general objects and follows general object detection paradigm to localize scene text by regressing the text box locations, but troubled by the arbitrary-orientation and large aspect ratios of scene text. The second one segments text regions directly, but mostly needs complex post processing. In this paper, we present a method that combines the ideas of the two types of methods while avoiding their shortcomings. We propose to detect scene text by localizing corner points of text bounding boxes and segmenting text regions in relative positions. In inference stage, candidate boxes are generated by sampling and grouping corner points, which are further scored by segmentation maps and suppressed by NMS. Compared with previous methods, our method can handle long oriented text naturally and doesn't need complex post processing. The experiments on ICDAR2013, ICDAR2015, MSRA-TD500, MLT and COCO-Text demonstrate that the proposed algorithm achieves better or comparable results in both accuracy and efficiency. Based on VGG16, it achieves an F-measure of 84.3% on ICDAR2015 and 81.5% on MSRA-TD500.

1. Introduction

Scene text detection must handle arbitrary orientations, extreme aspect ratios, and ambiguous boundaries. The proposed detector combines corner localization with position-sensitive segmentation to generate accurate boxes and achieves strong accuracy and efficiency on public benchmarks.

  • Scene text is difficult because text regions can be arbitrarily oriented, vary greatly in aspect ratio, and appear as characters, words, or lines.
  • The method detects four text-region corners and predicts position-sensitive maps instead of directly regressing boxes or using text/non-text maps.The corner points are top-left, top-right, bottom-right, and bottom-left.
  • Candidate boxes are generated by sampling and grouping detected corners, then filtered using segmentation information.
  • 84.3% F-measure on ICDAR2015, 81.5% on MSRA-TD500, and 72.4% on MLT were reported, with processing above 10.4 512x512 images per second.
  • The approach combines object detection and segmentation in an end-to-end trainable and evaluable detector.

2. Related Work

Prior work mainly regresses text boxes or segments text regions with substantial post-processing. This method instead forms arbitrary-oriented boxes from corner points and uses position-sensitive maps to score proposals.

  • 2.1. Regression Based Text Detection: Regression-based detectors adapt general object detection with long or rotated anchors, but directly regress text boxes or segments.
  • 2.1. Regression Based Text Detection: The proposed method localizes corner points and generates text boxes by sampling and grouping those points rather than regressing boxes directly.
  • 2.2. Segmentation Based Text Detection: Segmentation-based detectors predict text-related maps but typically require post-processing to obtain text bounding boxes.
  • 2.2. Segmentation Based Text Detection: Position-sensitive segmentation maps both predict text regions and score proposals, avoiding the complex post-processing used by earlier segmentation methods.
  • 2.3. Corner Point Based General Object Detection: Compared with earlier corner-point detectors, this method uses segmentation to score candidate boxes and produces arbitrary-oriented text boxes.

3. Network

The network is a fully convolutional architecture combining a DSSD-adapted backbone, multi-layer corner detection, and position-sensitive segmentation. Corner representations and proposal scoring are designed for varied text geometry.

  • 3. Network: The network jointly performs feature extraction, corner detection, and position-sensitive segmentation in a fully convolutional architecture.
  • 3.1. Backbone: Its backbone adapts VGG16 with FPN/DSSD-style features to address large text-scale variation and complex scene backgrounds.
  • 3.2. Corner Detection: A rotated text rectangle is represented by four clockwise corner coordinates, while each corner is detected using a horizontal square whose side equals the rectangle’s short side.
  • 3.2. Corner Detection: For each corner type and default box, the prediction branches output existence scores and coordinate offsets; four corner types are used by default.
  • 3.2. Corner Detection: The architecture uses multiple feature layers for corner detection, while the segmentation predictor shares features with the corner-detection branch.
  • 3.3. Position-Sensitive Segmentation: Position-sensitive segmentation divides a text box into grid bins and predicts whether pixels belong to each relative-position bin.

4. Training and Inference

Training converts text boxes into corner-point and position-sensitive segmentation labels, jointly optimizes these tasks, and assembles scored candidate boxes during inference.

  • Label Generation: Ground-truth text boxes are represented as rotated rectangles with four ordered corner points.The corners are ordered top-left, top-right, bottom-right, and bottom-left.
  • Label Generation: Corner-point labels use horizontal squares whose side length equals the rotated rectangle's short side.The four corner points are represented using the rectangle R derived from each ground-truth text box.
  • Label Generation: Position-sensitive segmentation divides each rotated rectangle into a 2 × 2 grid and assigns its bins to four pixel-wise masks.The masks encode text regions according to relative position rather than only text versus non-text.
  • Optimization: Corner detection and position-sensitive segmentation are trained jointly with confidence, localization, and segmentation losses.The losses are normalized by positive default-box and segmentation-pixel counts; default balancing factors are λ1 = 1 and λ2 = 10.
  • Sampling and Grouping: Inference retains corner points with scores above 0.5, applies NMS, and groups relative corner-point sets into candidate rotated boxes.Candidate boxes are formed by sampling compatible corner pairs and filtering violations of positional, side-length, and short-side constraints.
  • Scoring: Candidate boxes are scored from position-sensitive segmentation maps using rotated ROI average pooling, then low-score boxes are filtered at τ = 0.6.The pooling layer splits each rotated box into g × g bins, averages pixels within each bin, and averages the bin means.

5. Experiments

The method is evaluated on five public benchmarks spanning horizontal, oriented, long-line, multilingual, and large-scale scene text detection. These datasets provide varied settings for comparison with state-of-the-art methods.

  • Datasets: Experiments cover ICDAR2015, ICDAR2013, MSRA-TD500, MLT, and COCO-Text.The benchmarks include oriented, horizontal, long-line, multilingual, and large-scale scene text settings.
  • Datasets: SynthText contains about 800000 synthetic images and supplies word-level labels for pre-training.
  • Datasets: ICDAR2015 contains 1000 training images and 500 testing images with word-level quadrangle annotations.
  • Datasets: ICDAR2013 focuses on horizontal scene text and contains 229 training images and 233 testing images.
  • Datasets: MSRA-TD500 targets arbitrarily oriented long text lines using 300 training images and 200 test images with text-line annotations.
  • Datasets: MLT emphasizes multi-oriented, multi-script, and multilingual scene text across 7200 training, 2000 validation, and 9000 test images.

5.2. Implementation Details

The model is pre-trained on SynthText, fine-tuned on target datasets, optimized with Adam, and evaluated using a single NMS post-processing step. Training and inference use fixed image and hardware configurations.

  • Training: The model is pre-trained on SynthText and then fine-tuned on other datasets except COCO-Text.
  • Training: Adam optimization uses a fixed learning rate of 1e−4, with one SynthText pre-training epoch and dataset-dependent fine-tuning iterations.
  • Data Augmentation: Input patches are randomly sampled using SSD-style augmentation and resized to 512 × 512.
  • Post Processing: NMS is the only post-processing step, using a threshold of 0.3.
  • Implementation: Training uses batch size 24 on four GPUs, while evaluation uses batch size 1 on one GPU.

5.3. Detecting Oriented Text

Experiments on ICDAR2015 and ICDAR2013 assess oriented and horizontal text detection under single- and multi-scale settings. The method achieves strong accuracy while retaining efficient inference speed.

  • ICDAR2015: ICDAR2015 evaluation uses 500 fine-tuning epochs, vertical-text rotation augmentation during the last 15 epochs, and 768 × 1280 test inputs.
  • ICDAR2015: 84.3% F-measure on ICDAR2015 with multi-scale inputs exceeds the current best result by 3.3%.Single-scale evaluation achieves 80.7% F-measure, compared with 53.3% for the matched box-regression baseline.
  • ICDAR2015: The matched baseline comparison is 53.3% versus 80.7% F-measure, with only slight additional time cost.
  • ICDAR2013: 85.8% F-measure on ICDAR2013 at single scale is slightly below the highest result, while multi-scale evaluation reaches 88.0%.The single-scale speed is 10.4 FPS, faster than most compared methods.

5.5. Detecting Long Oriented Text Line

The method is evaluated for long and arbitrarily oriented text lines on MSRA-TD500, with examples spanning multiple benchmarks. It achieves strong precision, recall, and F-measure on this task.

  • MSRA-TD500: MSRA-TD500 evaluation uses HUST-TR400 as additional training data because MSRA-TD500 has only 300 training images.
  • Qualitative Results: Figure 6 presents detection examples from ICDAR2015, ICDAR2013, MSRA-TD500, MLT, and COCO-Text.
  • MSRA-TD500: Test images are input at 768 × 768 with τ set to 0.65.
  • MSRA-TD500: 87.6% recall, 76.2% precision, and 81.5% F-measure are reported on MSRA-TD500.The F-measure exceeds the previous best result of 77.0%.
  • MSRA-TD500: The method surpasses previous methods on MSRA-TD500 and is reported as more capable of detecting arbitrarily oriented long text.

5.6. Detecting Multi-Lingual Text

The method was evaluated on multilingual text in MLT and on COCO-Text for generalization, achieving strong benchmark results and ranking first in one challenge evaluation.

  • The method outperformed all competing methods on MLT by at least 3.1%.Testing used single-scale 768 × 768 images after fine-tuning a model pretrained on SynthText.
  • 42.5% F-measure was achieved on COCO-Text without additional training, outperforming competitors.The model was fine-tuned on ICDAR2015 and evaluated using COCO-Text V1.1 annotations.
  • The method ranked first among public results in the ICDAR2017 Robust Reading Challenge on COCO-Text.At an IoU threshold of 0.75, it exceeded other methods by a large margin.

5.8. Limitations

The method has difficulty separating extremely close text instances and detecting curved text under the reported training conditions.

  • Extremely close text instances may be predicted as a single instance when position-sensitive segmentation fails.The paper identifies this as a limitation and references Figure 7.
  • Curved text detection is weak because the training set contains few curved samples.This limitation is also illustrated in Figure 7.

6. Conclusion

The paper presents a detector that combines corner-point localization with position-sensitive segmentation and evaluates it across several scene-text benchmarks. The reported results demonstrate effectiveness and robustness for oriented, horizontal, long-oriented, and multilingual text.

  • The detector localizes text through corner-point detection and position-sensitive segmentation.The method was evaluated on public benchmarks covering oriented, horizontal, long-oriented, and multilingual text.
  • The reported benchmark performances demonstrate the method's effectiveness and robustness.
  • The authors identify constructing an end-to-end OCR system as future work.
Loading 1802.08948v2…