PaperlayerView exact revision on arXiv

Source-linked AI summary

Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spotting

Minghui Liao, Guan Pang, Jing Huang, Tal Hassner, Xiang Bai

arXiv:2007.09482v1cs.CV

TL;DR

Arbitrary-shape scene text spotters commonly rely on RPNs whose anchors and rectangular proposals struggle with extreme shapes and neighboring instances. Mask TextSpotter v3 replaces RPN with an anchor-free SPN and uses hard RoI masking, achieving stronger robustness across rotation, shape, and aspect-ratio benchmarks.

  • Problem

    RPN-based arbitrary-shape text spotters use manually designed anchors and axis-aligned proposals that struggle with extreme aspect ratios, irregular shapes, and neighboring instances.

  • Method

    Mask TextSpotter v3 uses an anchor-free Segmentation Proposal Network for polygonal proposals and hard RoI masking to suppress neighboring text and background noise.

  • Results

    The method surpasses state-of-the-art results by 21.9% on Rotated ICDAR 2013 and 5.9% on Total-Text, while achieving state-of-the-art performance on MSRA-TD500.

  • Takeaways & Limitations

    Accurate polygonal proposals and masked RoI features improve robustness to rotations, extreme aspect ratios, irregular shapes, and nearby text.

  • Takeaways & Limitations

    Recognition still experiences minor performance disturbance at some extreme rotation angles, such as 90°, because judging text-sequence direction is difficult.

Abstract

from arXiv · show

Recent end-to-end trainable methods for scene text spotting, integrating detection and recognition, showed much progress. However, most of the current arbitrary-shape scene text spotters use region proposal networks (RPN) to produce proposals. RPN relies heavily on manually designed anchors and its proposals are represented with axis-aligned rectangles. The former presents difficulties in handling text instances of extreme aspect ratios or irregular shapes, and the latter often includes multiple neighboring instances into a single proposal, in cases of densely oriented text. To tackle these problems, we propose Mask TextSpotter v3, an end-to-end trainable scene text spotter that adopts a Segmentation Proposal Network (SPN) instead of an RPN. Our SPN is anchor-free and gives accurate representations of arbitrary-shape proposals. It is therefore superior to RPN in detecting text instances of extreme aspect ratios or irregular shapes. Furthermore, the accurate proposals produced by SPN allow masked RoI features to be used for decoupling neighboring text instances. As a result, our Mask TextSpotter v3 can handle text instances of extreme aspect ratios or irregular shapes, and its recognition accuracy won't be affected by nearby text or background noise. Specifically, we outperform state-of-the-art methods by 21.9 percent on the Rotated ICDAR 2013 dataset (rotation robustness), 5.9 percent on the Total-Text dataset (shape robustness), and achieve state-of-the-art performance on the MSRA-TD500 dataset (aspect ratio robustness). Code is available at: https://github.com/MhLiao/MaskTextSpotterV3

1 Introduction

Scene text spotting must handle varied orientations, aspect ratios, and shapes, but RPN-based proposals can struggle with extreme or irregular text and merge neighboring instances. Mask TextSpotter v3 addresses these issues with anchor-free polygonal proposals and hard RoI masking, improving robustness across challenging benchmarks.

  • Scene text spotting combines text detection and recognition, requiring both localization and character-sequence prediction.
  • Rotation, aspect-ratio, and shape robustness are necessary because scene text varies in orientation, script-dependent length, and geometric form.
  • RPNs use manually designed axis-aligned anchors that poorly match extreme aspect ratios and can merge densely positioned neighboring text instances.
  • SPN replaces RPN with anchor-free polygonal proposals that represent arbitrary shapes and support hard RoI masking to suppress neighboring text or background noise.
  • 21.9% on Rotated ICDAR 2013, 5.9% on Total-Text, and state-of-the-art performance on MSRA-TD500 demonstrate improved robustness across rotation, shape, and aspect-ratio challenges.
  • The paper identifies SPN, hard RoI masking, and improved benchmark robustness as its three principal contributions.

2 Related work

Related work progresses from separate detection and recognition systems toward unified arbitrary-shape scene text spotters. Mask TextSpotter v3 builds on this line while addressing inaccurate RPN proposals through accurate polygonal proposals and RoI masking.

  • Scene text spotting methods include two-stage systems with separately trained detection and recognition networks and end-to-end systems integrating both tasks.
  • Mask TextSpotter v1 introduced end-to-end arbitrary-shape spotting using Mask R-CNN detection and character segmentation for recognition.
  • Mask TextSpotter v2 extended v1 with spatial attention, while other methods used mask-based recognition, character grouping, or centerline-based local regions.
  • Unlike prior RPN-based mask approaches that may produce inaccurate mask maps, Mask TextSpotter v3 uses accurate proposals and hard RoI masking for densely oriented or curved text.

3 Methodology

Mask TextSpotter v3 replaces RPN proposals with anchor-free polygonal proposals from an SPN, then applies hard RoI masking to reduce interference from neighboring text and background.

  • Architecture: Mask TextSpotter v3 combines an SPN, Fast R-CNN refinement, segmentation modules, and recognition modules in one pipeline.The SPN provides polygonal proposals while the remaining modules refine detection, segment instances and characters, and recognize text.
  • Segmentation Proposal Network: The SPN uses fused multi-scale features and segmentation masks to generate proposals without manually designed anchors.Its U-Net structure produces a segmentation map whose connected regions are expanded into proposals.
  • Segmentation Proposal Network: Polygonal SPN proposals represent extreme-aspect-ratio, densely oriented, and irregularly shaped text instances more accurately than axis-aligned proposals.The proposals are formed from contours of text regions, supporting arbitrary-shape text lines.
  • Hard RoI masking: Hard RoI masking multiplies RoI features by binary polygon masks to suppress background noise and neighboring text instances.The masked features are then used by the detection and recognition modules, reducing interference from content outside each polygon.
  • Optimization: The training objective sums SPN, Fast R-CNN, and mask-related losses, with both latter loss weights set to 1.0.The SPN uses dice loss, while the mask-related loss includes text-instance segmentation, character segmentation, and spatial-attention decoder terms.

4 Experiments

The experiments evaluate robustness across rotations, aspect ratios, shapes, and small text instances using standard benchmarks and compare qualitative results with Mask TextSpotter v2.

  • Evaluation: The evaluation measures robustness to rotations, aspect ratios, shapes, and small text instances across standard scene text benchmarks.The study also includes an ablation of hard RoI masking.
  • Datasets: The Rotated ICDAR 2013 benchmark tests rotation robustness by rotating test images and annotations through angles from 15° to 90°.Evaluation uses ICDAR 2015 protocols because ICDAR 2013 supports only axis-aligned bounding boxes.
  • Implementation details: The experiments use the same training data and settings as Mask TextSpotter v2 for fair comparison, with broader rotation augmentation otherwise applied.Images are randomly rotated within [−90°, 90°], while v2 originally uses [−30°, 30°]; both use the same augmentation for RoIC13 experiments.
  • Implementation details: Training uses SynthText pre-training followed by fine-tuning on a mixture of SynthText and four scene-text datasets.Optimization runs for 250k iterations with dataset sampling ratio 2 : 2 : 2 : 1 : 1 per mini-batch.

4.3 Rotation robustness

Mask TextSpotter v3 is evaluated for rotation robustness on RoIC13 through detection and end-to-end recognition across multiple rotation angles, with qualitative and quantitative comparisons against prior methods.

  • Mask TextSpotter v3 successfully handles densely oriented text instances that Mask TextSpotter v2 fails to detect and recognize.
  • Detection task: At 30°, 45°, and 60°, Mask TextSpotter v3 maintains more stable detection performance than Mask TextSpotter v2.The largest detection gap occurs at 45°.
  • Detection task: 26.8%, 18.0%, and 22.0% are Mask TextSpotter v3’s gains over Mask TextSpotter v2 in Precision, Recall, and F-measure at 45°.
  • End-to-end recognition task: Mask TextSpotter v3 surpasses Mask TextSpotter v2 by more than 19.2% in F-measure for end-to-end recognition at 45° and 60°.
  • End-to-end recognition task: At 45°, Mask TextSpotter v3 achieves 22.1, 21.0, and 21.9 performance gains over Mask TextSpotter v2 in the reported end-to-end recognition results.
  • The reported rotation robustness is attributed to SPN-generated accurate proposals and hard RoI masking that excludes neighboring text instances.

4.4 Aspect ratio robustness

Mask TextSpotter v3 is evaluated on MSRA-TD500 for extreme aspect-ratio text lines and on Total-Text for varied shapes, outperforming Mask TextSpotter v2 in both settings.

  • Aspect ratio robustness: MSRA-TD500 evaluates aspect-ratio robustness on text lines with extreme aspect ratios using detection only because recognition annotations are unavailable.
  • Aspect ratio robustness: 9.3% is Mask TextSpotter v3’s performance gain over Mask TextSpotter v2 on MSRA-TD500.Mask TextSpotter v3 also outperforms state-of-the-art methods designed for text-line detection.
  • Shape robustness: On Total-Text, Mask TextSpotter v3 obtains more accurate detection and recognition, especially for irregular shapes and large spaces between neighboring characters.
  • Shape robustness: 5.9% is Mask TextSpotter v3’s F-measure improvement over Mask TextSpotter v2 without a lexicon on Total-Text.

4.6 Small text instance robustness

Mask TextSpotter v3 improves robustness to small text in low-resolution images and performs strongly under generic-lexicon evaluation. Its hard RoI masking also achieves the best result among tested polygon-based masking variants.

  • Small text instance robustness: Mask TextSpotter v3 outperforms Mask TextSpotter v2 on all IC15 tasks with different lexicons.The IC15 challenges are low resolution and small text instances.
  • Lexicon robustness: 7.1% and 9.0% are the margins by which Mask TextSpotter v3 outperforms TextDragon with the generic lexicon.The generic 90k-word lexicon is described as more meaningful and challenging for real-world applications than strong or weak lexicons.
  • Lexicon robustness: The attention-based recognizer can learn language knowledge, making Mask TextSpotter v3 less dependent on strong-lexicon correction than TextDragon.TextDragon uses a CTC-based recognizer whose character prediction is more independent.
  • Ablation study: Direct-hard hard RoI masking achieves the best performance among the four tested combinations of polygon application and masking type.The comparison varies direct versus indirect use and soft versus hard masks.

5 Conclusion

Mask TextSpotter v3 is an end-to-end trainable arbitrary-shape scene text spotter using accurate polygonal proposals from an SPN. Experiments demonstrate robustness across rotation, aspect-ratio, shape, and small-text challenges.

  • Conclusion: Mask TextSpotter v3 is an end-to-end trainable arbitrary-shape scene text spotter that introduces SPN-generated accurate polygonal proposals.The method is more robust to rotated and irregularly shaped text than previous arbitrary-shape spotters using RPNs.
  • Conclusion: Experiments on Rotated ICDAR 2013, MSRA-TD500, and Total-Text demonstrate robustness to rotations, extreme aspect ratios, and shape variations.Results on IC15 additionally show robustness in detecting and recognizing small text instances.
  • Conclusion: Results on IC15 show that Mask TextSpotter v3 is robust in detecting and recognizing small text instances.

A Methodology details

The SPN segmentation prediction module uses convolution, batch normalization, and de-convolution operators. Its tabulated notation defines kernel size, stride, and padding.

  • Methodology details: The SPN segmentation prediction module includes convolution, batch normalization, and de-convolution operators.
  • Methodology details: In the module notation, k, s, and p denote kernel size, stride, and padding, respectively.

B Rotation robustness

The rotation-robustness appendix reports qualitative and quantitative results for Rotated ICDAR 2013. Detection results use precision, recall, and F-measure under a specified evaluation protocol and rotating-angle settings.

  • Rotation robustness: Qualitative and quantitative Rotated ICDAR 2013 results are reported in Fig. 8, Table B, and Table 7.
  • Rotation robustness: Table 6 reports Rotated ICDAR 2013 detection results using precision, recall, and F-measure under the ICDAR 2015 evaluation protocol.The table also reports rotating-angle settings and identifies the evaluation conditions for CharNet and Mask TextSpotter v2.

C Ablation study

The ablation study compares direct versus indirect and soft versus hard RoI masking, including qualitative and quantitative evaluations on Rotated ICDAR 2013. Direct-hard masking achieves the best performance by more strictly blocking background noise and neighboring text instances.

  • RoI masking is categorized as direct or indirect, depending on whether the segmentation or binary map is used directly or through additional layers.
  • Qualitative Rotated ICDAR 2013 results are shown across columns with rotation angles of 15°, 30°, 45°, 60°, 75°, and 90°.
  • Soft masks use probability values from [0, 1], whereas hard masks use binary polygon values of 0 or 1.
  • The experiments compare an original RoI-feature baseline with Direct-soft, Direct-hard, and Indirect-soft masking strategies.
  • Direct masking outperforms indirect masking, and hard masking outperforms soft masking in the ablation results.
  • Direct-hard masking achieves the best performance because it more strictly blocks background noise and neighboring text instances.
Loading 2007.09482v1…