Source-linked AI summary
Arbitrary Shape Scene Text Detection with Adaptive Text Region Representation
Xiaobing Wang, Yingying Jiang, Zhenbo Luo, Cheng-Lin Liu, Hyunsoo Choi, Sungjin Kim
TL;DR
Scene text detection must handle complex and irregular shapes, especially curved text, beyond the capabilities of many methods for horizontal and oriented text. The paper combines text proposals with RNN-based adaptive boundary-point refinement, achieving state-of-the-art results across five benchmarks and strong performance on curved and multi-oriented text. Future work considers corner-point detection and end-to-end recognition for arbitrary-shaped text.
Problem
Scene text detection remains challenging for irregular shapes such as curved text, although many methods address horizontal and oriented text.
Method
The method uses a Text-RPN for proposals and an RNN refinement network that predicts boundary-point pairs with adaptive length until a stop label.
Results
The method achieves state-of-the-art performance across five benchmarks and outperforms existing methods on CTW1500 while achieving Hmean 78.5% on TotalText.
Takeaways & Limitations
Adaptive boundary-point representation supports detection of horizontal, oriented, and arbitrary-shaped scene text, including curved text.
Takeaways & Limitations
Future work identifies corner-point detection and end-to-end recognition for arbitrary-shaped scene text as remaining improvements.
Abstract
from arXiv · showhide
Scene text detection attracts much attention in computer vision, because it can be widely used in many applications such as real-time text translation, automatic information entry, blind person assistance, robot sensing and so on. Though many methods have been proposed for horizontal and oriented texts, detecting irregular shape texts such as curved texts is still a challenging problem. To solve the problem, we propose a robust scene text detection method with adaptive text region representation. Given an input image, a text region proposal network is first used for extracting text proposals. Then, these proposals are verified and refined with a refinement network. Here, recurrent neural network based adaptive text region representation is proposed for text region refinement, where a pair of boundary points are predicted each time step until no new points are found. In this way, text regions of arbitrary shapes are detected and represented with adaptive number of boundary points. This gives more accurate description of text regions. Experimental results on five benchmarks, namely, CTW1500, TotalText, ICDAR2013, ICDAR2015 and MSRATD500, show that the proposed method achieves state-of-the-art in scene text detection.
1. Introduction
Scene text detection is important for extracting text from complex natural images, but variations in scene appearance make detection challenging. The paper proposes a two-stage method with adaptive boundary-point representation for arbitrary-shaped text and reports strong results across five benchmarks.
- Scene text detection supports applications including translation, blind person assistance, shopping, robotics, smart cars, and education.
- Detection precedes recognition in end-to-end text recognition because text regions must be located before their content can be retrieved.
- Complex backgrounds and variations in font, size, color, language, illumination, and orientation make scene text detection challenging.
- The method first extracts text proposals with a Text-RPN, then verifies and refines them using a refinement network.
- A recurrent representation predicts boundary-point pairs until stopping, enabling arbitrary-shaped text regions to use adaptive numbers of points.
2. Related work
Earlier scene text detectors used sliding-window, connected-component, bounding-box, segmentation, or combined strategies, but fixed point representations remain poorly matched to varied text shapes. This paper therefore uses a two-stage detector with adaptive, RNN-learned boundary representations.
- Traditional sliding-window and connected-component methods classify image patches or character candidates as text or non-text using bottom-up processing.
- Deep learning scene text detectors include bounding-box regression, segmentation, and combined approaches.
- Bounding-box regression methods include one-stage detectors that estimate regions directly and two-stage detectors that generate and refine proposals.
- Two-stage methods usually perform better than one-stage methods, motivating the paper's proposal-generation and refinement design.
- Fixed point counts are inefficient across shapes: 14 points may underspecify long curved text while exceeding the needs of horizontal or oriented text.
- The proposed adaptive representation uses an RNN to predict different numbers of pairwise boundary points, avoiding the need for pixel-wise segmentation.
3. Methodology
The method uses a two-stage detector with Text-RPN proposals followed by refinement that predicts adaptive polygonal text representations. An LSTM outputs pairwise boundary points and stop/continue labels, while classification, box regression, and polygon NMS complete detection.
- Text proposal and refinement: Text-RPN generates text proposals and CNN feature maps, which a refinement network then verifies and refines into final detections.The refinement network includes text/non-text classification, bounding box regression, and adaptive text region representation branches.
- Adaptive text region representation: Adaptive point counts address the mismatch between fixed representations for simple texts and complex curved or long text regions.Existing methods use fewer points for horizontal or oriented text and fixed 14-point polygons for curved text, which may be insufficient or wasteful depending on shape.
- Adaptive text region representation: Pairwise points on the top and bottom boundaries are learned directionally from one end of a text region to the other.This representation is presented as easier to learn than unordered corner points and can describe regions with different shapes precisely.
- Sequence-based representation learning: An LSTM predicts boundary-point coordinates at each time step because different text regions require sequences of different lengths.The LSTM receives ROI-pooled proposal features, and its coordinate regression uses point coordinates as targets.
- Sequence-based representation learning: Each LSTM time step predicts two boundary points and a stop/continue label through separate coordinate-regression and classification branches.Prediction continues until the stop label is produced, yielding an adaptive number of boundary points for each text region.
- Training objective: The refinement objective combines text classification, bounding-box regression, boundary-point regression, and stop/continue classification losses.The loss terms are balanced with λ1, λ2, and λ3, each set to 1 in the proposed method.
4. Experiments
Experiments evaluate the method on five benchmarks spanning horizontal, oriented, curved, irregular, multilingual, and long straight scene texts. Ablations and comparisons show benefits from adaptive representation, strong benchmark performance, and faster processing than competing arbitrary-shape detectors.
- Ablation studies: SE-VGG16 performs better than VGG16 on both CTW1500 and ICDAR2015.The backbone comparison evaluates the two networks on both datasets.
- Ablation studies: 80.2% recall versus 76.4% shows adaptive text region representation outperforming fixed 14-point representation on CTW1500.The ablation supports adaptive representation as more suitable for texts of arbitrary shapes.
- State-of-the-art comparisons: 80.1% Hmean on CTW1500 exceeds CTD, CTD+TLOC, and TextSnake at 69.5%, 73.4%, and 75.6%, while TotalText reaches 78.5% Hmean.These datasets contain challenging multi-oriented and curved texts.
- State-of-the-art comparisons: 91.7% Hmean on ICDAR2013 matches Mask Textspotter, while 87.6% on ICDAR2015 trails FOTS at 88.0%.The ICDAR2015 result uses single-scale input and a single mode; FOTS combines detection and recognition during training.
- State-of-the-art comparisons: 83.6% Hmean on MSRA-TD500 is better than all other methods, and the method runs much faster than Mask Textspotter and TextSnake.The speed advantage is attributed to avoiding pixel-wise prediction and requiring less computation.
- Qualitative results: Qualitative results cover arbitrarily oriented or curved text, different languages, nonuniform illumination, and word- or sentence-level text lengths.Results are illustrated across all five benchmarks.
5. Conclusion
The paper concludes that its adaptive text-region representation supports scene text detection across horizontal, oriented, and arbitrary-shape texts. It reports especially strong performance on CTW1500 and MSRA-TD500, while identifying corner-point detection and end-to-end recognition as future improvements.
- Conclusion: The proposed method detects horizontal, oriented, and arbitrary-shape scene texts using adaptive text region representation.The conclusion summarizes experiments on five benchmarks.
- Conclusion: The method particularly outperforms existing methods on CTW1500 and MSRA-TD500.These benchmarks are described as typical of curved texts and multi-oriented texts, respectively.
- Future work: Future work includes corner-point detection for easier training annotations and end-to-end recognition for arbitrary-shape scene text.These are stated as proposed directions for improving detection and extending the system toward recognition.