Source-linked AI summary
Shape Robust Text Detection with Progressive Scale Expansion Network
Xiang Li, Wenhai Wang, Wenbo Hou, Ruo-Ze Liu, Tong Lu, Jian Yang
TL;DR
Natural-scene text detection remains challenging because text and background vary substantially in shape, appearance, orientation, scale, illumination, and occlusion. PSENet addresses this with multi-scale kernels and progressive expansion, surpassing the previous state-of-the-art by 6.37% on SCUT-CTW1500 while performing competitively on regular text benchmarks.
Problem
Natural-scene text detection faces substantial variation in text and background appearance, including shape, color, font, orientation, scale, illumination, and occlusion.
Method
PSENet predicts multiple segmentation masks for text-instance kernels at different scales and progressively expands them from the smallest kernels to complete instances.
Results
6.37% absolute improvement over the previous state-of-the-art on SCUT-CTW1500, with better or comparable performance on ICDAR 2015 and ICDAR 2017 MLT.
Takeaways & Limitations
Progressive expansion supports arbitrary-shaped text detection and separates text instances that are close together or partially intersected.
Takeaways & Limitations
The paper leaves open whether the scale expansion algorithm can be trained jointly with the network end-to-end.
Abstract
from arXiv · showhide
The challenges of shape robust text detection lie in two aspects: 1) most existing quadrangular bounding box based detectors are difficult to locate texts with arbitrary shapes, which are hard to be enclosed perfectly in a rectangle; 2) most pixel-wise segmentation-based detectors may not separate the text instances that are very close to each other. To address these problems, we propose a novel Progressive Scale Expansion Network (PSENet), designed as a segmentation-based detector with multiple predictions for each text instance. These predictions correspond to different `kernels' produced by shrinking the original text instance into various scales. Consequently, the final detection can be conducted through our progressive scale expansion algorithm which gradually expands the kernels with minimal scales to the text instances with maximal and complete shapes. Due to the fact that there are large geometrical margins among these minimal kernels, our method is effective to distinguish the adjacent text instances and is robust to arbitrary shapes. The state-of-the-art results on ICDAR 2015 and ICDAR 2017 MLT benchmarks further confirm the great effectiveness of PSENet. Notably, PSENet outperforms the previous best record by absolute 6.37\% on the curve text dataset SCUT-CTW1500. Code will be available in https://github.com/whai362/PSENet.
1 Introduction
Natural-scene text detection must handle diverse shapes and crowded instances that challenge both quadrangular detectors and ordinary segmentation methods. PSENet addresses these limitations with multi-scale kernels and progressive expansion, achieving strong benchmark results.
- Natural-scene text detection remains challenging because text and backgrounds vary widely in shape, color, font, orientation, scale, illumination, and occlusion.
- Bounding-box regression methods locate rectangular or quadrangular text but struggle with arbitrary shapes such as curved text.
- PSENet predicts multiple segmentation kernels for each text instance, with shared centers and progressively increasing scales.The kernels are expanded from minimal to maximal scales using a three-step BFS-based procedure.
- Minimal kernels provide separated starting regions, while gradual expansion recovers complete text shapes and supports accurate instance localization.The paper also motivates multiple scales as providing smoother supervision and avoiding the need for excessive numbers of scales.
- 6.37% absolute improvement over the previous best result is reported on SCUT-CTW1500, while performance is better or comparable on ICDAR 2015 and ICDAR 2017 MLT.
- The paper contributes PSENet and its progressive scale expansion algorithm for arbitrary-shape detection and separation of closely adjacent text instances.
2 Related Work
Text detection methods progressed from horizontal and rotated rectangles toward irregular quadrangles, but natural scenes also contain curved and other non-quadrangular text shapes.
- Earlier text detectors adapted object-detection pipelines to horizontal text and later incorporated orientation for arbitrarily oriented instances.
- Corner localization enabled irregular quadrangular boxes, but quadrangular representations remain insufficient for many natural-scene text shapes.
3 Proposed Method
PSENet combines multi-scale segmentation with progressive expansion to detect arbitrary-shaped text while separating adjacent instances. It generates shrinking kernels and expands them from minimal to complete text regions, using scale-specific supervision and a combined loss.
- 3.1 Overall Pipeline: PSENet fuses multi-level features and predicts n segmentation masks with progressively increasing kernel scales.S1 represents minimal kernels, Sn represents complete text instances, and expansion produces final detections R.
- 3.2 Progressive Scale Expansion Algorithm: The progressive expansion algorithm initializes connected components from minimal kernels, then merges eligible pixels through successive masks before extracting labeled final predictions.The procedure expands kernels from S1 through S2 and S3, grouping pixels by their kernel labels.
- 3.2 Progressive Scale Expansion Algorithm: Breadth-First Search expands adjacent text pixels from multiple kernels, assigning conflicted pixels to the first kernel that reaches them.The algorithm processes a queue and treats each segmentation mask as a successive expansion stage.
- 3.3 Label Generation: Training labels are generated by shrinking each original text polygon into sequential masks using Vatti clipping and scale ratios increasing linearly from m to 1.The margin between the original polygon pn and a shrunk polygon pi is computed from area and perimeter quantities.
- 3.4 Loss Function: PSENet optimizes a weighted combination of complete-instance and shrunk-instance losses, using Dice similarity and OHEM for difficult text-versus-background patterns.Lc handles complete text segmentation, while Ls handles shrunk instances and ignores non-text pixels outside complete text regions.
- Implementation Details: The reported implementation uses six scales from 0.5 to 1.0, loss weight λ=0.7, and an OHEM negative-positive ratio of 3.Training also ignores regions marked DO NOT CARE and applies rescaling, flipping, and rotation augmentation.
4 Experiment
Experiments evaluate PSENet on quadrangular and curve-text benchmarks, with ablations examining kernel scales and qualitative comparisons testing separation, shape robustness, and efficiency.
- Benchmark evaluation: PSENet is evaluated on ICDAR 2015, ICDAR 2017 MLT, and SCUT-CTW1500 against state-of-the-art methods.The experiments cover oriented quadrangular text and arbitrarily shaped curve text.
- Ablation study: Increasing the number of kernel scales n raises ICDAR 2015 F-measure until performance levels off at n ≥6, while n = 1 provides a traditional semantic-segmentation baseline.The ablation fixes m = 0.5 and varies n from 2 to 10; the single-mask setting shows a large performance gap.
- Ablation study: The minimal scale m requires balance: large values hinder separation of adjacent text instances, whereas small values can split text lines and destabilize training.This ablation uses n = 6 and varies m from 0.1 to 0.9 on ICDAR 2015.
- Quadrangular text: 85.51% recall is achieved on ICDAR 2015, while ICDAR 2017 MLT reaches 72.45% F-measure and exceeds FOTS by 5.2%.PSENet also reports satisfactory efficiency on NVIDIA GTX 1080 Ti.
- Curve text: 81.17% F-measure on SCUT-CTW1500 surpasses the second-best record by 6.37%, with precision of 82.50% and recall of 79.89%.Qualitative comparisons show detection of complex curve instances and separation of closely spaced text.
- Qualitative comparisons: Qualitative examples show separation of nearby text instances across orientations, illumination conditions, colors, scales, and extreme curvature.On SCUT-CTW1500, some detections correspond to text instances absent from the groundtruth labels.
5 Conclusion and Future Work
PSENet detects arbitrarily shaped text by progressively expanding small kernels into complete instances, while separating nearby or partially intersected text. Experiments show superior scene-text detection performance, with end-to-end training remaining future work.
- PSENet progressively expands small kernels into large, complete text instances using multiple semantic segmentation maps.
- The method is robust to arbitrary shapes and distinguishes text instances that are very close or partially intersected.
- Experiments on scene text detection benchmarks demonstrate superior performance for the proposed method.
- Future work includes training the scale expansion algorithm jointly with the network in an end-to-end manner.