PaperlayerView exact revision on arXiv

Source-linked AI summary

Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes

Minghui Liao, Pengyuan Lyu, Minghang He, Cong Yao, Wenhao Wu, Xiang Bai

arXiv:1908.08207v1cs.CV

TL;DR

Scene text spotting must jointly detect and recognize text despite limitations in prior end-to-end training and irregular text handling. Mask TextSpotter uses two-dimensional instance and semantic segmentation with spatial attention, achieving state-of-the-art results across detection, recognition, and spotting benchmarks. Its main scope boundary is that character segmentation still requires character-level annotations and post-processing when used as described.

  • Problem

    Prior text spotters face incomplete end-to-end training, difficult recognition optimization, and limited support for irregular text shapes.

  • Method

    Mask TextSpotter combines instance segmentation for detection with two-dimensional character segmentation and spatial attention for recognition.

  • Results

    State-of-the-art performance is reported across scene text detection, recognition, and end-to-end text recognition benchmarks, including horizontal, oriented, and curved text.

  • Takeaways & Limitations

    The two-dimensional representation supports reading text of varied shapes while integrating local character information with global sequence prediction.

  • Takeaways & Limitations

    Character segmentation requires character-level annotations and a specially designed post-processing algorithm to obtain text sequences.

Abstract

from arXiv · show

Unifying text detection and text recognition in an end-to-end training fashion has become a new trend for reading text in the wild, as these two tasks are highly relevant and complementary. In this paper, we investigate the problem of scene text spotting, which aims at simultaneous text detection and recognition in natural images. An end-to-end trainable neural network named as Mask TextSpotter is presented. Different from the previous text spotters that follow the pipeline consisting of a proposal generation network and a sequence-to-sequence recognition network, Mask TextSpotter enjoys a simple and smooth end-to-end learning procedure, in which both detection and recognition can be achieved directly from two-dimensional space via semantic segmentation. Further, a spatial attention module is proposed to enhance the performance and universality. Benefiting from the proposed two-dimensional representation on both detection and recognition, it easily handles text instances of irregular shapes, for instance, curved text. We evaluate it on four English datasets and one multi-language dataset, achieving consistently superior performance over state-of-the-art methods in both detection and end-to-end text recognition tasks. Moreover, we further investigate the recognition module of our method separately, which significantly outperforms state-of-the-art methods on both regular and irregular text datasets for scene text recognition.

1 INTRODUCTION

Scene text spotting jointly detects and recognizes text in natural images, but prior systems face training and shape-handling limitations. Mask TextSpotter addresses these issues with two-dimensional segmentation and spatial attention for arbitrary-shaped text.

  • Scene text detection and recognition are complementary, yet separate pipelines may produce sub-optimal performance because recognition depends on detections and recognition can remove false positives.
  • Earlier end-to-end spotters still rely on curriculum, alternating, or ground-truth-region training rather than fully smooth end-to-end optimization.Their recognition components require accurate locations, while LSTM or CTC objectives are harder to optimize than general CNNs.
  • Prior methods mainly handle horizontal or oriented text, whereas real-world text can also be curved and irregular.
  • Mask TextSpotter detects text with instance segmentation and recognizes it through semantic segmentation and spatial attention in two-dimensional space.This design targets arbitrary-shaped text and avoids relying exclusively on one-dimensional sequence recognition.
  • SAM globally predicts word label sequences using word-level annotations and complements character segmentation, which locally predicts character labels from pixels.Compared with the conference version, this reduces the need for character-level training annotations and mitigates heuristic character grouping.
  • 10.5 percent: Mask TextSpotter outperforms previous top performers on ICDAR2015 end-to-end recognition with the generic lexicon.The paper also reports state-of-the-art detection and text spotting across English and multilingual datasets, plus standalone recognition gains.

2 RELATED WORK

Related work progresses from handcrafted and deep-learning pipelines toward integrated text spotting, while most methods remain restricted in shape handling. Mask TextSpotter uses instance segmentation and combines local character and global attention-based recognition.

  • Earlier scene text spotters used handcrafted features, including character detection with random ferns followed by pictorial-structure grouping with a fixed lexicon.
  • Deep-learning systems improved detection and recognition through CNN-based character, proposal, and word-classification components, while later methods adopted sequence-to-sequence recognition.
  • Recent methods integrate detection and recognition in end-to-end networks because the two tasks are complementary.
  • Scene text detection research increasingly addresses multi-oriented and arbitrary-shaped text using segmentation, linking, direct regression, symmetry, and axis-based representations.
  • Mask TextSpotter differs by detecting scene text through instance segmentation, enabling arbitrary-shaped text detection.
  • Its recognition design integrates character segmentation with spatial attention to reduce character-level annotation needs and improve robustness to text shapes.

3.1 Architecture

Mask TextSpotter uses an FPN-backed, Mask R-CNN-inspired architecture with proposal, box-regression, and mask branches. The mask branch jointly supports text-instance segmentation, character segmentation, and sequence recognition.

  • Mask TextSpotter comprises an FPN backbone, RPN proposal generator, Fast R-CNN box-regression branch, and mask branch.
  • RPN generates text proposals, after which RoI features feed Fast R-CNN and the mask branch during training.The branches produce candidate boxes, instance maps, character maps, and text sequences.
  • The FPN with ResNet-50 fuses features at different resolutions to build semantic maps across text scales.
  • RPN uses five anchor stages with areas 32^2, 64^2, 128^2, 256^2, and 512^2 pixels, plus aspect ratios 0.5, 1, and 2.These settings are intended to cover varied text sizes and aspect ratios.
  • Fast R-CNN performs classification and regression on 7 × 7 RoI features to provide accurate detection boxes.
  • The mask branch performs text-instance segmentation, character segmentation, and text sequence recognition.

3.2 Text Instance and Character Segmentation

Given fixed-size RoI features, the mask branch produces a text-instance map and character segmentation maps from shared convolutional features. This supports localization independent of text-instance shape and pixel-level character prediction.

  • A 16 × 64 RoI feature passes through four 3 × 3 convolutional layers and a 2 × 2 deconvolutional layer before branching.
  • A one-channel text-instance map provides accurate text-region localization regardless of the text-instance shape.
  • The character-segmentation module directly uses shared features to generate character maps with shape N_s × 32 × 128.N_s represents the number of character classes or channels in the output maps as described in the passage.

3.3 Spatial Attentional Module (SAM)

SAM decodes character sequences directly from two-dimensional feature maps using iterative spatial attention and recurrent prediction. It addresses character segmentation's annotation, post-processing, and ordering limitations while supporting varied text shapes.

  • Motivation: SAM removes character segmentation's need for character-level annotations, specialized grouping, and explicit character ordering.These limitations arise because segmentation maps do not directly provide the text sequence.
  • Spatial Attention with RNNs: SAM directly decodes a two-dimensional feature map into a character sequence, avoiding one-dimensional encoding of spatial features.The module operates iteratively for T steps and predicts character classes y = (y1, ..., yT).
  • Spatial Attention with RNNs: At each step, SAM uses the input feature map, the previous hidden state, and the previously predicted character class.The previous hidden state is expanded into a feature map before attention weights are computed.
  • Spatial Attention with RNNs: Attention weights are learned over the two-dimensional spatial grid from the hidden-state map, input feature map, and trainable parameters.The resulting weights have shape (Hp, Wp).
  • Spatial Attention with RNNs: The attention weights produce a glimpse from the original feature map, which is combined with the previous character embedding as the RNN input.The RNN cell receives this cascaded input together with the previous hidden state.
  • Spatial Attention with RNNs: A linear transformation and softmax compute the conditional probability of the character class at each decoding step.The decoder uses 37 classes, including 36 alphanumeric classes and one end-of-sequence symbol.

3.4 Standalone Recognition Model

The standalone recognition model combines pixel-level character segmentation with SAM-based sequence decoding. Its shared feature representation uses a ResNet-50 feature pyramid enhanced with multi-scale context and preserved resolution.

  • Feature Extraction: The model uses a ResNet-50 feature pyramid with Pyramid Pooling and dilated convolutions that preserve the resolution of the last two stages.Feature maps from the pyramid are up-sampled and concatenated into a shared map for both recognition modules.
  • Architecture: The standalone recognition model contains a character segmentation module and a Spatial Attention Module.The two modules provide complementary recognition mechanisms.
  • Character Segmentation: The character segmentation module predicts characters at pixel level and can use pixel voting to group and arrange them into a text sequence.This module requires a post-processing step to form the final sequence result.
  • Spatial Attention Module: SAM predicts text sequences from a two-dimensional representation in an end-to-end manner.It provides the second recognition route in the standalone model.

3.5 Label Generation

Label generation creates targets for region proposals, text-instance masks, character maps, and SAM sequence recognition. Polygon geometry is normalized to proposals, while character supervision is optional when annotations are unavailable.

  • Input Annotations: Training samples provide text polygons and, when available, character categories with character locations.These annotations generate targets for RPN, Fast R-CNN, and the mask branch.
  • Text Instance Targets: Text polygons are converted into minimal-area horizontal rectangles before RPN and Fast R-CNN targets are generated.The mask branch additionally receives text-instance and character target maps.
  • Text Instance Targets: For each positive proposal, polygon coordinates and character boxes are normalized relative to the proposal.The normalized geometry supports proposal-specific target generation.
  • Text Instance Targets: The text-instance target map is formed by drawing the normalized polygon on an empty mask and filling its region with 1.This produces a binary mask for the text region.
  • Character Targets: Character maps use shrunken character boxes labeled by category index, while pixels outside them are background and missing annotations are ignored.Character boxes are shrunk to one fourth of their original side lengths around fixed centers.
  • Sequence Targets: SAM receives word-level character-category sequences without character localization information.Thus sequence supervision does not require character boxes.

3.6 Optimization

Optimization combines proposal, classification, text-instance, character-segmentation, and sequence-recognition objectives in a multi-task loss. The label-generation design supplies separate spatial and sequence supervision for the mask branch.

  • Multi-task Objective: The model uses a multi-task loss because it includes multiple learning tasks.The objective combines the proposal and recognition-related components.
  • Mask Loss: The mask loss combines text-instance segmentation, character segmentation, and sequence recognition losses.These components are denoted Lins, Lseg, and Lseq.
  • Component Losses: The text-instance loss is average binary cross-entropy, while character segmentation uses a weighted spatial soft-max loss.The weighting balances character-class positives against the background class.
  • Sequence Loss: The sequence loss is computed from the decoder's conditional character probabilities over the sequence-label length.The probability p(yt) is defined by the sequence decoder.
  • Loss Weights: The loss weights α1 and α2 and β1 are set to 1.0, while β2 is set to 0.2.These values are empirical settings for the multi-task objective.

3.7 Inference

Inference uses detector-filtered proposals to generate text regions, character maps, and sequences, then decodes recognition outputs into final text results. A weighted edit distance optionally selects the best lexicon match using probability-dependent operation costs.

  • Fast R-CNN proposals replace training-time RPN proposals during inference because they are more accurate.
  • Nonmaximum suppression removes redundant candidate boxes before proposals enter the mask branch.
  • The mask branch generates text instance maps, character maps, and text sequences, with polygons obtained from text-region contours.
  • Decoding: Pixel voting binarizes the background map at 0.75, assigns each connected region its highest-mean character class, and orders characters from left to right.
  • Decoding: SAM sequences use beam search rather than greedy decoding, retaining the top k probabilities with k = 6.
  • Weighted Edit Distance: Weighted edit distance resolves lexicon matches by making deletion, insertion, and replacement costs depend on character probabilities instead of assigning all costs equally.

4 EXPERIMENTS

Experiments evaluate Mask TextSpotter across horizontal, oriented, curved, multilingual, and standalone recognition benchmarks. The method achieves strong detection and recognition results, with particular advantages for irregular text and large or absent lexicons.

  • Datasets: Experiments cover four English datasets and one multilingual dataset spanning horizontal, oriented, and curved text, plus standalone scene-text recognition benchmarks.The datasets include ICDAR2013, ICDAR2015, COCO-Text, Total-Text, and MLT; SynthText supplies training data.
  • Overall results: Mask TextSpotter achieves state-of-the-art detection results and remains comparable with leading methods for word spotting and end-to-end recognition.The comparison includes single-scale evaluation against methods using multiple scales.
  • Oriented text: 87% f-measure on ICDAR2015 is comparable to the previous state of the art, while generic-lexicon word spotting and end-to-end recognition improve by 8.6 and 10.5 percentage points.The method also achieves the best recall among the methods reported in Table 1 and remains comparable with strong or weak lexicons.
  • Curved text: On curved text, the method improves detection by 3.9 percentage points over [12] and end-to-end recognition by at least 28.5% over TextBoxes.The authors attribute the recognition advantage to representing text sequences in two-dimensional space rather than as one-dimensional sequences.
  • Recognition module: The spatial attention module raises end-to-end recognition by 12.4 percentage points without a lexicon and 5.6 points with a lexicon versus the conference version.The paper reports slight detection improvements as well and links the recognition gains to SAM's global reading view.
  • Speed: Inference runs at 3.8, 3.1, and 2.0 FPS for input scales of 720×1280, 1000×1778, and 1600×2844, respectively.At 720×1280, detection takes about 0.20 seconds and recognition about 0.06 seconds on average.
  • Standalone recognition: The standalone recognition model outperforms ASTER on all 12 tasks across seven benchmarks, including gains of 5.1% on SVTP and 9.0% on CUTE.The largest improvements occur on irregular text benchmarks involving perspective and curved shapes.
  • Experimental conclusions: The experiments conclude that the method handles varied shapes, depends less on lexicons, benefits from SAM, and supports strong recognition of irregular text.The summary also reports reduced dependence on real-world character-level annotations compared with the conference version.

5 CONCLUSION

Mask TextSpotter uses a two-dimensional representation for both detection and recognition, enabling end-to-end text spotting for irregular text. It achieves state-of-the-art results across horizontal, oriented, and curved-text benchmarks, while detection remains its main efficiency bottleneck.

  • Mask TextSpotter uses a two-dimensional representation for both text detection and recognition, rather than one-dimensional sequence prediction.
  • The framework is easy to train and can read irregular text, including curved text.
  • Mask TextSpotter achieves state-of-the-art results on scene text detection, scene recognition, and end-to-end text recognition benchmarks.The benchmarks cover horizontal, oriented, and curved text.
  • The detection stage is the most time-consuming part of the proposed method, motivating future efficiency improvements.
Loading 1908.08207v1…