Source-linked AI summary
Shape Robust Text Detection with Progressive Scale Expansion Network
Wenhai Wang, Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Gang Yu, Shuai Shao
TL;DR
Scene-text detection must handle arbitrary-shaped text and separate nearby instances, challenges that regression- and segmentation-based approaches do not both resolve. PSENet expands small segmentation kernels into complete text regions. On CTW1500, it reports a 6.6% absolute improvement over state-of-the-art results, with a real-time model achieving 74.3% at 27 FPS.
Problem
Regression-based approaches struggle with arbitrary-shaped text, while segmentation-based approaches can merge nearby instances into one false detection.
Source: §1 · p. 2Method
PSENet predicts multiple-scale kernels for each instance and progressively expands minimal-scale kernels into complete text regions.
Source: §1 · p. 2Results
On CTW1500, PSENet outperforms state-of-the-art results by 6.6% absolute, while its real-time model achieves 74.3% at 27 FPS.
Source: §1 · p. 2Takeaways & Limitations
PSENet detects arbitrary-shaped text and can separate nearby or partially intersected instances on scene-text benchmarks.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Scene text detection has witnessed rapid progress especially with the recent development of convolutional neural networks. However, there still exists two challenges which prevent the algorithm into industry applications. On the one hand, most of the state-of-art algorithms require quadrangle bounding box which is in-accurate to locate the texts with arbitrary shape. On the other hand, two text instances which are close to each other may lead to a false detection which covers both instances. Traditionally, the segmentation-based approach can relieve the first problem but usually fail to solve the second challenge. To address these two challenges, in this paper, we propose a novel Progressive Scale Expansion Network (PSENet), which can precisely detect text instances with arbitrary shapes. More specifically, PSENet generates the different scale of kernels for each text instance, and gradually expands the minimal scale kernel to the text instance with the complete shape. Due to the fact that there are large geometrical margins among the minimal scale kernels, our method is effective to split the close text instances, making it easier to use segmentation-based methods to detect arbitrary-shaped text instances. Extensive experiments on CTW1500, Total-Text, ICDAR 2015 and ICDAR 2017 MLT validate the effectiveness of PSENet. Notably, on CTW1500, a dataset full of long curve texts, PSENet achieves a F-measure of 74.3% at 27 FPS, and our best F-measure (82.2%) outperforms state-of-art algorithms by 6.6%. The code will be released in the future.
1. Introduction
PSENet addresses the difficulty of detecting arbitrarily shaped text and separating nearby instances. It uses progressively expanded kernels to recover complete text regions and reports benchmark results including a 6.6% absolute improvement on CTW1500.
1. Introduction
- Regression-based approaches struggle with arbitrary-shaped text, while segmentation-based approaches can merge nearby instances into one false detection.Regression methods commonly represent text with rectangles or quadrangles; segmentation methods classify pixels but have difficulty separating close instances.Source: §1 · p. 2
- PSENet combines pixel-level segmentation with progressive scale expansion to locate arbitrary-shaped text and distinguish adjacent instances.It predicts kernels at multiple scales, then uses breadth-first search to expand minimal-scale kernels until the complete text instances are reached.Source: §1 · p. 2
- Minimal-scale kernels have boundaries far apart, making them easier to separate, but they do not cover complete text instances and must be expanded.The paper describes progressive expansion as a simple, efficient way to recover complete instances from the smaller kernels.Source: §1 · p. 2
- On CTW1500, PSENet outperforms state-of-the-art results by 6.6% absolute, while its real-time model achieves 74.3% at 27 FPS.The introduction also reports promising performance on ICDAR 2015 and ICDAR 2017 MLT.Source: §1 · p. 2
2. Related Work
Related work divides CNN-based scene-text detectors into regression-based and segmentation-based methods. The paper contrasts their limitations on curved or nearby text with approaches designed for irregular shapes and its own scale-expansion procedure.
2. Related Work
- Regression-based methods use rectangles or quadrangles and may need complex anchor design and multiple stages, while potentially falling short on curve text.Examples include TextBoxes [19], EAST [42], RRPN [28], and RRD.Source: §2 · p. 2
- Segmentation-based methods are mainly inspired by FCNs [25], with examples predicting text-region and orientation heatmaps [38], localizing corners for irregular quadrangles [27], and pixel connections to separate nearby text instances [4].Source: §2 · p. 2
- TextSnake [26] addresses curve text, but uses time-consuming, complicated post-processing; PSENet's scale expansion is described as one clean, efficient step.The related-work discussion also notes SPC-Net [?] as an instance-segmentation approach using context to detect arbitrary shapes while suppressing false positives.Source: §2 · p. 3
3. Proposed Method
PSENet fuses multilevel features to predict text masks at multiple kernel scales, then expands minimal kernels to complete text shapes. The method also specifies scale-specific training labels and a loss combining complete and shrunk text regions.
3.1. Overall Pipeline
- PSENet fuses low- and high-level feature maps, then projects the fused representation into branches that predict segmentation masks at different kernel scales.The smallest-scale mask supplies minimal kernels, while the largest-scale mask represents the original text instances; expansion produces final detections.Source: §3.1 · p. 3
3.2. Network Design
- The network concatenates four 256-channel feature maps after upsampling them to a shared scale, forming a 1024-channel fused map before producing n segmentation outputs.The fused map is reduced to 256 channels before n convolution-upsampling-sigmoid branches generate the masks.Source: §3.2 · p. 3
3.3. Progressive Scale Expansion Algorithm
- A BFS-inspired expansion starts from connected components in the minimal kernels and iteratively absorbs adjacent text pixels across increasingly larger masks.Conflicted pixels are assigned to the first kernel that reaches them; the authors state these boundary conflicts do not affect final detections or performance.Source: §3.3 · pp. 3–4, Figure 4 · p. 4
3.4. Label Generation
- Training masks are generated by shrinking each original text polygon with the Vatti clipping algorithm, with scale ratios increasing linearly from m to 1.The scale ratios are determined by n and the minimal ratio m.Source: §3.4 · pp. 4–5, Figure 5 · p. 5
3.5. Loss Function
- The loss combines complete-instance and shrunk-instance terms, uses Dice for sparse text pixels, applies OHEM to the complete-text loss, and masks non-text pixels for the shrunk-text loss.OHEM is used to better distinguish text-like patterns such as fences and lattices.Source: §3.5 · p. 5
4. Experiment
Experiments evaluate PSENet on four text-detection benchmarks, with ablations, state-of-the-art comparisons, and speed analysis. The reported results include gains on curve, oriented, and multilingual text.
4.1. Datasets
- The benchmarks cover long curve text, mixed horizontal, multi-oriented and curve text, quadrangle-annotated text, and nine-language scenes.The datasets are CTW1500, Total-Text, ICDAR 2015, and ICDAR 2017 MLT, respectively.Source: §4.1 · pp. 5–6
4.2. Implementation Details
- Training used ImageNet-pretrained ResNet for IC17-MLT and either scratch training or IC17-MLT fine-tuning for the other datasets, with image augmentation and OHEM.No extra data was used to train on IC17-MLT; blurred regions marked DO NOT CARE were ignored.Source: §4.2 · p. 6
4.3. Ablation Study
- Minimal-scale kernels alone performed poorly as final detections; F-measure fell when minimal scale was too large or too small, while gains from more kernels leveled off at n ≥5.Large kernels hinder separation of close instances; very small kernels can split text lines and impede convergence.Source: §4.3 · p. 6, Figure 6 · p. 6
- Increasing the ResNet backbone depth from 50 to 152 improved IC17-MLT performance from 70.8% to 72.2%, an absolute gain of 1.4%.The comparison used the same setting across backbone depths.Source: §4.3 · p. 6, Table 1 · p. 7
4.4. Comparisons with State-of-the-Art Methods
- 82.2% F-measure on CTW1500 was 6.6% higher than TextSnake, and PSENet surpassed the other counterparts without external data.The reported CTW1500 result was also 8.8% higher than CTD+TLOC.Source: §4.4 · p. 7, Table 2 · p. 7
- 80.87% F-measure on Total-Text outperformed state-of-the-art methods by over 2.47%.PSENet achieved 84.02% precision and 77.96% recall on the same benchmark.Source: §4.4 · p. 7, Table 3 · p. 7
- 85.69% F-measure on ICDAR 2015 surpassed state-of-the-art results by more than 3%, while IC17-MLT reached 72.13%, 5.3% above state of the art.The IC17-MLT evaluation used a doubled original image; the corresponding tables report single-scale results.Source: §4.4 · p. 7, Table 4 · p. 7, Table 5 · p. 8
4.5. Speed Analyze
- 27 FPS was nearly real-time with ResNet18, and PSENet surpassed EAST and CTD+TLOC in both speed and performance.On CTW1500, reducing the output map to one-quarter resolution raised speed from 3.9 to 8.4 FPS while F-measure decreased from 82.2% to 79.9%.Source: §4.5 · p. 8, Table 6 · p. 8
5. Conclusion and Future Work
PSENet detects arbitrarily shaped scene text by progressively expanding small kernels into complete text instances. The authors identify end-to-end training of the expansion algorithm and applying it to crowded instance-segmentation tasks as future directions.
5. Conclusion and Future Work
- PSENet progressively expands detected areas from small kernels to complete instances through multiple semantic segmentation maps, supporting arbitrary shapes and separating close or partially intersecting text.Source: §5 · p. 8
- Future work includes investigating whether the expansion algorithm can be trained end-to-end with the network.Source: §5 · p. 8
- The authors propose exploring progressive scale expansion for instance-level segmentation, especially benchmarks with crowded object instances.They also state that they are cleaning the code for release.Source: §5 · p. 8
7. Supplementary Materials for Shape Robust Text Detection with Progressive Scale Expansion Network
Supplementary experiments examine multi-kernel reconstruction, portability to PSPNet, and qualitative behavior on curve-text and multilingual benchmarks.
7.1. The Advantage of Multiple Kernels
- Three-kernel reconstruction alleviates flaws in two-kernel reconstruction, while PSE time complexity remains O(W × H) as kernel count increases.Figure 8 compares two-kernel and three-kernel reconstruction.Source: §7.1 · p. 9, Figure 8 · p. 9
7.2. Applying PSENet on Other Semantic Segmentation Framework
- The PSPNet-based PSENet-like method is competitive on CTW1500 but uses 3.7G vs 2.9G GPU memory and takes 289ms vs 118ms forward time versus original PSENet.Both methods were trained from scratch.Source: §7.2 · p. 9, Table 7 · p. 9
7.3. More Comparisons on CTW1500
- PSENet locates some unmarked instances and separates curve texts with extreme curvature from nearby instances.The comparisons are on CTW1500.Source: §7.3 · p. 9
7.4. More Detected Examples on Total Text, ICDAR 2015 and ICDAR 2017 MLT
- PSENet detects arbitrary-shaped and closely adjacent text, varied orientations, multilingual text, and instances under complex illumination, colors, and scales.These examples come from Total-Text, ICDAR 2015, and ICDAR 2017 MLT.Source: §7.4 · p. 9