PaperlayerView exact revision on arXiv

Source-linked AI summary

Scene Text Detection with Supervised Pyramid Context Network

Enze Xie, Yuhang Zang, Shuai Shao, Gang Yu, Cong Yao, Guangyao Li

arXiv:1811.08605v1cs.CV

TL;DR

Scene text detectors can produce false positives in complex natural scenes, while text varies widely in shape and appearance. The paper proposes an FPN- and instance-segmentation-based detector with semantic guidance, Text Context, and Re-Score mechanisms. It reports higher performance than previous state-of-the-art methods across four benchmarks, while adding little computation.

  • Problem

    False positives and arbitrary-shaped text remain difficult for scene text detection in complex natural scenes.

  • Method

    The method combines Mask R-CNN, FPN, instance segmentation, semantic segmentation guidance, a Text Context module, and a Re-Score mechanism.

  • Results

    92.1% F-measure is achieved on ICDAR2013, 87.2% on ICDAR2015, 74.1% on ICDAR2017MLT, and 82.9% on Total-Text, outperforming previous state-of-the-art methods.

  • Takeaways & Limitations

    The approach detects text with arbitrary shapes and suppresses false positives while adding little computation compared with the baseline.

  • Takeaways & Limitations

    Future work includes integrating Re-Score into the network end-to-end and developing more efficient mobile text-detection networks.

Abstract

from arXiv · show

Scene text detection methods based on deep learning have achieved remarkable results over the past years. However, due to the high diversity and complexity of natural scenes, previous state-of-the-art text detection methods may still produce a considerable amount of false positives, when applied to images captured in real-world environments. To tackle this issue, mainly inspired by Mask R-CNN, we propose in this paper an effective model for scene text detection, which is based on Feature Pyramid Network (FPN) and instance segmentation. We propose a supervised pyramid context network (SPCNET) to precisely locate text regions while suppressing false positives. Benefited from the guidance of semantic information and sharing FPN, SPCNET obtains significantly enhanced performance while introducing marginal extra computation. Experiments on standard datasets demonstrate that our SPCNET clearly outperforms start-of-the-art methods. Specifically, it achieves an F-measure of 92.1% on ICDAR2013, 87.2% on ICDAR2015, 74.1% on ICDAR2017 MLT and 82.9% on Total-Text.

Introduction

Scene text detection remains difficult because natural scenes generate false positives and contain text with varied shapes and orientations. The paper proposes a semantic-information-guided detector with context modeling and re-scoring, reporting improved benchmark performance.

  • Motivation: False positives and arbitrary-shaped text remain central challenges in natural-scene text detection.Challenges arise from varied foreground and background appearance, text shape, color, font, orientation, scale, illumination, and occlusion.
  • Motivation: Mask R-CNN can lack global context, causing classification errors for background objects with text-like textures.Its region-of-interest classification does not use surrounding-scene information, while false positives can correlate with contextual objects.
  • Approach: The proposed detector uses mask outputs to locate text areas, enabling detection of horizontal, oriented, and curved text.The approach is designed to detect text with arbitrary shapes under semantic-information guidance.
  • Approach: Text Context and Re-Score mechanisms are introduced to suppress false positives and improve classification using semantic information.The Text Context module guides detection with semantic segmentation, while Re-Score combines segmentation activation values with classification scores.
  • Results: 92.1% F-measure is achieved on ICDAR2013, 87.2% on ICDAR2015, 74.1% on ICDAR2017MLT, and 82.9% on Total-Text.The method outperforms previous state-of-the-art algorithms across horizontal, oriented, multilingual, and curved-text benchmarks.

Related Work

Prior scene text detectors mainly use segmentation-based or regression-based approaches, but many remain limited in false-positive suppression and arbitrary-shape flexibility. The paper combines semantic guidance with Mask R-CNN to address these limitations.

  • Existing approaches: Previous deep-learning scene text detectors are broadly divided into segmentation-based and regression-based methods.Segmentation-based methods derive text regions from pixel-level predictions, whereas regression-based methods predict text boxes or geometric representations.
  • Segmentation-based methods: Segmentation-based methods use varied labels and often require time-consuming post-processing while achieving unsatisfying performance.Examples include text blocks, character links, text kernels, and progressive scale expansion.
  • Regression-based methods: Regression-based and general detection methods adapt object-detection architectures for large-aspect-ratio, oriented, or long text.Examples include modified SSD, FCN-based prediction, and separate classification and regression features.
  • Paper positioning: The proposed pipeline uses deep supervised semantic information to guide Mask R-CNN, suppress false positives, and detect arbitrary-shaped text end-to-end.The method combines instance and semantic segmentation and reports a performance gap over previous methods on several benchmarks.

Proposed method

The proposed pipeline is built on Mask R-CNN and adds a Text Context module plus a post-processing Re-Score mechanism. These components target false-positive suppression through contextual semantic information and score adjustment.

  • Proposed method: The pipeline consists of a Text Context module and a post Re-Score mechanism built on Mask R-CNN.The method also includes a loss-function design described alongside these components.
  • Proposed method: The Text Context module and Re-Score mechanism are designed to further suppress false positives.The Re-Score mechanism is applied during post-processing, while the Text Context module provides contextual semantic guidance.

Mask R-CNN

The architecture uses Mask R-CNN as its instance-segmentation basis and augments an FPN-based detector with global text segmentation. Proposals feed classification and mask branches at different resolutions.

  • Mask R-CNN: Mask R-CNN provides the instance-segmentation basis and a strong baseline for text detection.The paper motivates this choice by identifying Mask R-CNN as state of the art in instance segmentation.
  • Ground-truth construction: Text polygons define instance masks, while their minimum bounding horizontal rectangles define bounding boxes.Pixels inside polygons are treated as text and pixels outside as non-text; global binary maps are generated similarly.
  • Architecture: The architecture combines FPN, RPN, R-CNN, mask prediction, and global text segmentation branches.RoI-Align resizes proposals to 7×7 for the R-CNN branch and 14×14 for mask prediction, while global segmentation operates across FPN stages.

Text Context Module

The Text Context Module combines pyramid attention and fusion to add global semantic context to Mask R-CNN text detection. Its supervised text/non-text segmentation guides feature activation and complements ROI-based classification, improving discrimination between text and background.

  • Text Context Module: The Text Context Module comprises Pyramid Attention and Pyramid Fusion sub-modules and produces text segmentation as output.
  • Pyramid Attention Module: Global text segmentation branches after FPN generate pixel-level text/non-text saliency maps for each FPN layer.
  • Pyramid Attention Module: Channel-wise softmax followed by exponential activation enlarges the response gap between text and non-text regions before saliency-based feature activation.
  • Pyramid Fusion Module: Pyramid Fusion combines detection features with deeply supervised semantic features, making the network more discriminative for text versus non-text.
  • Pyramid Fusion Module: Semantic segmentation uses pixel and surrounding-pixel information, while detection classifies regions through ROIs, creating complementary branches.
  • Architecture: The architecture includes FPN, RPN, R-CNN, mask prediction, and global text segmentation branches; the segmentation map is upsampled for supervised loss.

Re-Score Mechanism

The Re-Score Mechanism replaces reliance on classification confidence alone by combining Mask R-CNN classification with activation from a global text segmentation map. This reduces false-positive confidence while supporting tilted text with strong segmentation responses.

  • Motivation: Standard inference ranks proposals by classification confidence before NMS, retaining the highest-confidence boxes for Mask R-CNN instance prediction.
  • Score construction: The fused score combines classification score (CS) and instance score (IS) for each predicted text instance.
  • Instance score: The instance score is computed from the mean activation of pixels covered by the projected text instance on the global text segmentation map.
  • Effect: The fused score reduces false-positive confidence because false-positive instances tend to respond more weakly on the segmentation map than text.
  • Effect: Tilted text can receive compensation for relatively low classification scores when its segmentation response produces a high instance score.

Loss Function Design

The network extends Mask R-CNN’s multitask loss with global text segmentation, enabling joint learning of text detection and semantic context from the same input.

  • Loss Function Design: The loss combines standard Mask R-CNN losses with an additional global text segmentation loss, Lgts.Lrpn, Lcls, Lbox, and Lmask remain the standard Mask R-CNN terms.
  • Loss Function Design: Lgts is a Softmax loss applied to the network’s output prediction p.
  • Loss Function Design: Multitask learning jointly trains text detection and global text segmentation end-to-end from the same input.Gradients from both tasks influence shared feature maps.

Experiments

Experiments evaluate the method across four benchmarks spanning horizontal, incidental, multilingual, multi-oriented, and curved text. Ablations indicate that the proposed modules improve precision and overall F-measure while adding little computation.

  • Experiments: The approach is evaluated on ICDAR2013, ICDAR2015, ICDAR2017 MLT, and Total-Text against state-of-the-art methods.
  • Experiments: The experiments use SynthText pretraining, Mask R-CNN-style augmentation, and polygon NMS post-processing.SynthText provides 800000 synthetic images for word-level pretraining; images are resized and randomly flipped during augmentation.
  • Experiments: The benchmarks cover horizontal text, incidental scene text, multilingual and multi-oriented text, and curved text.ICDAR2013 targets horizontal text; ICDAR2017 MLT covers multi-oriented, multi-scripting, and multilingual text; Total-Text targets curved text.
  • Ablation Study: The MLT ablation study evaluates the Text Context module and Re-Score mechanism against a baseline Mask R-CNN architecture.
  • Ablation Study: 4.1% higher precision with identical recall is achieved by the Text Context module relative to baseline.
  • Experiments: The qualitative results cover ICDAR2013, ICDAR2015, ICDAR2017, and Total-Text.
  • Ablation Study: 3.9% additional precision improvement from Re-Score yields a total 3.8% F-measure gain over baseline.The mechanism re-ranks text-instance scores during inference and is reported to suppress false positives.

Results on Scene Text Benchmarks

The proposed method achieves strong results across horizontal, oriented, multilingual, and curved scene-text benchmarks, surpassing prior methods. Its reported performance includes 74.1% F-measure on ICDAR2017 MLT, 92.1% on ICDAR2013, and results above prior art on ICDAR2015 and Total-Text.

  • Detecting MultiLingual Text: 74.1% F-measure on ICDAR2017 MLT surpasses all competing methods by at least 1.7%.The result uses merged outputs from two test scales in [720, 1920].
  • Detecting Oriented Text: The method surpasses state-of-the-art results by more than 1.5% on ICDAR2015 with single-scale testing.The experiments use the MLT-pretrained model and fine-tune it for 40 epochs.
  • Detecting Horizontal Text: 92.1% F-measure on ICDAR2013 establishes a state-of-the-art result for horizontal text detection.The method is designed for arbitrary shapes but also performs strongly on horizontal text.
  • Detecting Curved Text: The method surpasses prior art on Total-Text, demonstrating applicability to curved text detection without special modifications.EAST and SegLink results in the comparison are included only for reference because they were not fine-tuned on Total-Text.

Conclusion

The paper concludes that its end-to-end detector combines semantic segmentation guidance with instance segmentation to detect arbitrary-shaped text and reduce false positives. Shared convolutional features make the segmentation branch nearly cost-free, while benchmark results support the method’s effectiveness and generalization.

  • Conclusion: The detector uses semantic segmentation guidance and a Re-Score mechanism to alleviate false positives for predicted text instances.The framework is end-to-end trainable and combines instance segmentation with semantic segmentation.
  • Conclusion: The method detects text with arbitrary shapes through a shape-robust, end-to-end trainable framework.The paper presents this capability as part of its overall detector design.
  • Conclusion: Shared convolutional features make the text segmentation branch nearly cost-free.This design limits the additional computation associated with semantic guidance.
  • Conclusion: Results on different scene-text benchmarks demonstrate the approach’s effectiveness and generalization.The conclusion summarizes evidence across multiple benchmarks rather than a single dataset.
  • Conclusion: Future work includes integrating Re-Score into the network end-to-end and developing faster text detectors for mobile phones.The authors also propose exploring other multi-oriented or curved object-detection tasks.

Acknowlegement

The work was supported by the National Natural Science Foundation of China and acknowledges assistance from Mr. Mengxiao Lin.

  • Acknowlegement: The authors acknowledge funding from the National Natural Science Foundation of China under Grant 61771346.
  • Acknowlegement: The paper thanks Mr. Mengxiao Lin of the Megvii base-model group for his help.
Loading 1811.08605v1…