Source-linked AI summary
Text Perceptron: Towards End-to-End Arbitrary-Shaped Text Spotting
Liang Qiao, Sanli Tang, Zhanzhan Cheng, Yunlu Xu, Yi Niu, Shiliang Pu, Fei Wu
TL;DR
Irregular text localization may not provide regions that recognition can handle, and conventional pipelines limit joint optimization. Text Perceptron combines segmentation-based detection, STM, and sequence recognition in an end-to-end trainable framework. It reports competitive results on standard benchmarks and state-of-the-art results on irregular-text benchmarks.
Problem
Irregular text localization may not yield regions suitable for recognition, while conventional detection-recognition pipelines are not end-to-end trainable.
Source: §1 · p. 1Method
Text Perceptron combines segmentation-based detection, an STM that tunes fiducial points, and sequence recognition in an end-to-end trainable framework.
Source: §1 · p. 2Results
Text Perceptron reports competitive results on standard text benchmarks and state-of-the-art results in detection and end-to-end evaluations on popular irregular-text benchmarks.
Source: §5 · p. 8Takeaways & Limitations
On Total-Text and CTW1500, results demonstrate effectiveness for arbitrary-shaped text spotting, with end-to-end training markedly boosting spotting performance, especially recognition.
Source: §4.4 · p. 7Vertical text is rare in common datasets, and recognition can confuse it with horizontal or lying-down text, a difficulty shared by existing recognition algorithms.
Source: Failure Samples · p. 8
Abstract
from arXiv · showhide
Many approaches have recently been proposed to detect irregular scene text and achieved promising results. However, their localization results may not well satisfy the following text recognition part mainly because of two reasons: 1) recognizing arbitrary shaped text is still a challenging task, and 2) prevalent non-trainable pipeline strategies between text detection and text recognition will lead to suboptimal performances. To handle this incompatibility problem, in this paper we propose an end-to-end trainable text spotting approach named Text Perceptron. Concretely, Text Perceptron first employs an efficient segmentation-based text detector that learns the latent text reading order and boundary information. Then a novel Shape Transform Module (abbr. STM) is designed to transform the detected feature regions into regular morphologies without extra parameters. It unites text detection and the following recognition part into a whole framework, and helps the whole network achieve global optimization. Experiments show that our method achieves competitive performance on two standard text benchmarks, i.e., ICDAR 2013 and ICDAR 2015, and also obviously outperforms existing methods on irregular text benchmarks SCUT-CTW1500 and Total-Text.
1 Introduction
Irregular text localization can score well while still failing to yield recognition-ready regions, exposing an incompatibility between text detection and recognition in conventional pipelines.
1 Introduction
- Irregular layouts challenge quadrangle-based methods, while segmentation-based localization can achieve good IoU yet leave recognition challenges unresolved.Real-world text may be multi-oriented or curved.Source: §1 · p. 1
- Conventional pipelines crop detected text and use rectification, but require extra geometric supervision and computation and cannot use recognition errors to optimize detection.Even true-positive detections can produce clipped masks and incorrect recognition.Source: §1 · p. 1
- Text Perceptron combines segmentation-based detection, STM, and sequence recognition in an end-to-end trainable spotter.Its detector represents text with center, head, tail, and top-and-bottom boundary regions; STM tunes fiducial points using recognition feedback.Source: §1 · p. 2
- Experiments report competitive results on two regular-text benchmarks and results surpassing previous methods on two irregular-text benchmarks.The contributions describe the detector as order-aware and STM as differentiable, supporting joint detection and recognition optimization.Source: §1 · p. 2
2 Related Works
The related-work review surveys text detection and end-to-end spotting, including methods for irregular text and their remaining challenges.
2 Related Works
- The review covers recent advances in text detection and end-to-end text spotting.Source: §2 · p. 2
2.1 Text Detection
- Anchor-based detectors use rectangular proposals, while segmentation-based methods describe arbitrary shapes but can require postprocessing to separate instances.Prior approaches include long or quadrilateral boxes, rotated proposals, masks, boundary segmentation, and progressive expansion.Source: §2.1 · p. 2
2.2 Text Spotting
- Existing spotting pipelines often crop text for sequence recognition; end-to-end methods address joint optimization, but prior approaches struggle with arbitrary shapes or curved text.Character-by-character spotting can lose character context and require character-level annotations.Source: §2.2 · p. 2
3 Methodology
Text Perceptron combines order-aware segmentation, geometric regression, and STM-based TPS rectification with sequence recognition in one trainable spotter. Recognition gradients can also adjust detector-derived fiducial points.
3.1 Overview
- The overview combines a ResNet/FPN detector, STM-based differentiable TPS rectification, and a sequence recognizer in one trainable text-spotting framework.The recognizer may use a sequence-based method such as CRNN or an attention-based method.Source: §3.1 · p. 3
3.2 Text Detection Module
- Order-aware segmentation labels center, head, tail, and top/bottom boundary regions, with head and tail encoding potential reading order and boundaries separating instances.The detector also learns corner and boundary-offset regression tasks to provide position information for fiducial points.Source: Order-aware Semantic Segmentation · p. 3, Corner and Boundary Regressions · p. 3
- Corner regression predicts offsets from head and tail pixels to corresponding corners, while boundary-offset regression uses center-region pixels to locate nearby boundaries.The detector uses proximity regression for corners and can ignore or down-weight offsets to the larger side, helping describe text with large width-height ratios.Source: Corner and Boundary Regressions · p. 3
3.3 Shape Transform Module
- STM generates potential fiducial points from corner positions and detector boundary offsets, using a dichotomous strategy to place intermediate points along text boundaries.The described strategy is intended for arbitrary-shaped text, including seriously curved text and differing reading orders.Source: Fiducial Points Generation · pp. 4–5
- STM applies TPS to map irregular feature regions into regular regions, then sends the transformed regions to the sequence recognizer.Recognition differences are back-propagated through STM to adjust the feature region and its fiducial points.Source: Shape Transformation · p. 5, Dynamically Finetuning Fiducial Points · p. 5
3.4 End-to-End Training
- End-to-end training combines segmentation, corner regression, boundary-offset regression, and recognition losses, with weights tuned so fiducials shift from regression-led early to recognition-led late.The paper describes this soft loss-weight strategy as automatic tuning of the regression and recognition weights.Source: §3.4 · p. 5
4 Experiments
Experiments evaluate Text Perceptron on standard and irregular-text benchmarks, with ablations and visualizations probing its performance across text shapes and fiducial-point counts.
4.1 Datasets
- Evaluation uses SynthText 800k for pretraining and four benchmarks: focused IC13, perspective-rich IC15, multi-oriented or curved Total-Text, and line-level curved CTW1500.Total-Text uses word-level polygon annotations; CTW1500 uses 14-point line-level polygons.Source: §4.1 · pp. 5–6
4.3 Results on Standard Text Benchmarks
- On IC13, results are competitive across detection, end-to-end, and word-spotting tasks at 10.3 FPS; on IC15, end-to-end and word spotting reach state of the art for general lexicon cases.IC15 detection performance is described as comparable to irregular-text spotting methods.Source: Table 1 · p. 6, §4.3 · p. 6
4.2 Implementation Details
- The detector uses ResNet-50 and FPN features with attention-based recognition; detection and recognition are pretrained separately, then the whole network is jointly fine-tuned and tested at single scale.Fiducial-point counts are set to 4 for standard datasets and 14 for irregular datasets.Source: §4.2 · p. 6
4.4 Results on Irregular Text Benchmarks
- Text Perceptron surpasses prior detection F-measure results by 2.3% on Total-Text and 2.4% on CTW1500, and best end-to-end results by 15.7% on None and 6.5% on Full.The authors report no prior end-to-end CTW1500 result and provide their lexicon-free result there.Source: Table 2 · p. 7, Table 3 · p. 7, §4.4 · p. 7
4.5 Ablation Results of Fiducial Points
- For irregular benchmarks, detection and end-to-end F-measures rise with more fiducial points and stabilize when 2 × N ≥10; four points suffice for regular IC15.The examples show that few points may not cover an entire curve, while more points help STM rectify irregular instances; the method can generate any fixed number of points on text boundaries.Source: Table 4 · p. 7, §4.5 · p. 7
4.6 Visualization Results
- Visualizations show reading-order capture across curved, long-perspective, and vertical text; fiducial points support recognition after rectification, including turning vertical text into lying-down shapes.Missing head or tail boundaries filter out many text-like false positives, indicating those boundaries carry semantic information distinct from the center region.Source: Figure 6 · p. 7, §4.6 · pp. 7–8, Figure 7 · p. 8
- For overlapped text, inner-instance boundaries are often not fully recalled, and connecting center-text pixels can therefore fail to detect the inner instance.The authors describe this as a difficult case for segmentation-based detection methods.Source: Failure Samples · p. 8
- Vertical-text recognition remains difficult because the recognizer may not distinguish horizontal text from a lying-down vertical instance; such text is uncommon in common datasets.The authors note this as a common difficulty for existing recognition algorithms.Source: Failure Samples · p. 8
5 Conclusion
Text Perceptron is an end-to-end trainable spotter for arbitrary-shaped text. Its segmentation-based detector captures latent reading order, while a Shape Transform Module unites detection and recognition; experiments report competitive standard-benchmark results and state-of-the-art results on irregular-text benchmarks.
5 Conclusion
- The method achieves competitive results on standard text benchmarks and state-of-the-art detection and end-to-end results on popular irregular-text benchmarks.Source: §5 · p. 8
- The Shape Transform Module unites detection and recognition in one framework to support global optimization.Source: §5 · p. 8
- The segmentation-based detector distinguishes text instances and captures latent text reading-order information.Source: §5 · p. 8