Source-linked AI summary
Detecting Text in Natural Image with Connectionist Text Proposal Network
Zhi Tian, Weilin Huang, Tong He, Pan He, Yu Qiao
TL;DR
Natural-image text detection is difficult because text has large pattern variation, cluttered backgrounds, and less-defined boundaries than generic objects. The CTPN addresses this with fine-scale sequential proposals, vertical anchors, and an in-network recurrent mechanism, achieving state-of-the-art performance on five benchmarks at 0.14s/image.
Problem
Natural-image text detection requires accurate localization despite large text-pattern variation, cluttered backgrounds, and text lacking well-defined closed boundaries.
Method
CTPN detects text lines as sequences of fine-scale proposals using vertical anchors and an in-network recurrent mechanism directly on convolutional feature maps.
Results
CTPN achieves new state-of-the-art performance on five benchmarks with 0.14s/image running time.
Takeaways & Limitations
The integrated proposal and recurrent design enables accurate detection of highly challenging text with fewer false detections and no additional post-processing.
Takeaways & Limitations
Some challenging images were missed by ground-truth labeling, which may reduce the detector's measured precision.
Abstract
from arXiv · showhide
We propose a novel Connectionist Text Proposal Network (CTPN) that accurately localizes text lines in natural image. The CTPN detects a text line in a sequence of fine-scale text proposals directly in convolutional feature maps. We develop a vertical anchor mechanism that jointly predicts location and text/non-text score of each fixed-width proposal, considerably improving localization accuracy. The sequential proposals are naturally connected by a recurrent neural network, which is seamlessly incorporated into the convolutional network, resulting in an end-to-end trainable model. This allows the CTPN to explore rich context information of image, making it powerful to detect extremely ambiguous text. The CTPN works reliably on multi-scale and multi- language text without further post-processing, departing from previous bottom-up methods requiring multi-step post-processing. It achieves 0.88 and 0.61 F-measure on the ICDAR 2013 and 2015 benchmarks, surpass- ing recent results [8, 35] by a large margin. The CTPN is computationally efficient with 0:14s/image, by using the very deep VGG16 model [27]. Online demo is available at: http://textdet.com/.
Connectionist Text Proposal Network
The paper is authored by Zhi Tian, Weilin Huang, Tong He, Pan He, and Yu Qiao, and focuses on scene text detection using convolutional and recurrent networks with anchor mechanisms.
- Zhi Tian and Weilin Huang are listed among the paper’s authors.
- Tong He, Pan He, and Yu Qiao are also listed as authors.
- The paper’s keywords are scene text detection, convolutional networks, recurrent neural networks, and anchor mechanisms.
1 Introduction
The introduction frames natural-image text detection as a fine-grained localization problem and proposes CTPN to replace complex bottom-up pipelines with an end-to-end sequence-based detector.
- Motivation: Natural-image text detection is challenging because text patterns vary widely and backgrounds are highly cluttered.
- Motivation: Bottom-up methods typically combine character or stroke detection with filtering, line construction, and verification, making them complicated and less robust.
- Motivation: Text detection requires more accurate localization than generic object detection because complete text-line coverage is needed for comprehensive reading.
- CTPN: CTPN extends the RPN architecture to localize text sequences directly in convolutional maps while avoiding an additional costly CNN detection stage.
- CTPN: CTPN represents text lines as sequences of fine-scale proposals and jointly predicts each proposal’s vertical location and text/non-text score.
- CTPN: An in-network recurrent mechanism connects sequential proposals, enabling the detector to use text-line context for challenging text.
- Contributions: The unified model handles multi-scale and multilingual text in one process without post-filtering or refinement.
- Results: 0.88 F-measure on ICDAR 2013 exceeded 0.83 in, while 0.61 on ICDAR 2015 exceeded 0.54 in.
2 Related Work
Related work covers bottom-up scene-text methods based on character or stroke detection and CNN-based object-detection systems that generate and refine proposals.
- Text detection: Scene-text detection has been dominated by bottom-up methods using stroke or character detection.
- Text detection: These methods include connected-components approaches and sliding-window methods that use low-level visual properties to form character candidates.
- Object detection: CNN object-detection systems commonly generate object proposals with inexpensive features before applying a strong CNN for classification and refinement.
3 Connectionist Text Proposal Network
CTPN localizes text lines as sequences of fine-scale proposals on convolutional feature maps, combining vertical anchors, recurrent context, and side-refinement for accurate detection.
- Detecting Text in Fine-scale Proposals: CTPN detects text lines as sequences of fixed-width fine-scale proposals rather than as single whole objects.Each proposal generally represents a small text portion, such as strokes, character parts, or characters.
- Detecting Text in Fine-scale Proposals: Vertical anchors jointly predict each proposal’s vertical location and text/non-text score across varied text scales and aspect ratios.The detector uses fixed 16-pixel proposal widths and multiple vertical anchors at each convolutional feature-map location.
- Detecting Text in Fine-scale Proposals: CTPN uses shared convolutional computation from densely sliding windows over VGG16 feature maps to generate proposal predictions efficiently.A 3×3 window slides over conv5 features, whose total stride is 16 pixels and receptive field is 228×228 pixels.
- Recurrent Connectionist Text Proposals: An in-network recurrent connection links sequential proposals and uses context to reduce false detections and recover ambiguous or weak text patterns.The RNN processes sliding-window features sequentially, and its hidden state feeds the prediction layers.
- Side-refinement: Side-refinement estimates left and right offsets for the endpoint proposals, improving final text-line localization by about 2% on SWT and Multi-Lingual datasets.The offsets are predicted simultaneously by the model rather than computed through an additional post-processing step.
4 Experimental Results and Discussions
Across five benchmarks, CTPN achieves strong text-detection accuracy on challenging, multi-scale, and multilingual images while remaining computationally efficient. Recurrent connections improve F-measure substantially, although missed ground-truth annotations can reduce measured precision on extremely small text.
- Evaluation setup: CTPN was evaluated on five benchmarks using the datasets’ standard evaluation protocols.The benchmarks were ICDAR 2011, ICDAR 2013, ICDAR 2015, SWT, and Multilingual.
- Component analysis: The fine-scale text proposal network improved Faster R-CNN in both precision and recall by predicting sequences of fine-scale proposals.Faster R-CNN achieved a 0.75 F-measure, while FTPN provided more accurate and reliable localization.
- Component analysis: 0.88 F-measure: recurrent connections improved FTPN performance from 0.80 to 0.88 on ICDAR 2013.The recurrent mechanism also reduces false detections and helps recover highly ambiguous text.
- Efficiency: 0.14s/image: the complete CTPN detector runs efficiently on a single GPU, compared with 0.13s/image without recurrent connections.The recurrent mechanism adds marginal computation while providing a considerable performance gain.
- Qualitative results and limitations: CTPN handles multi-scale and multilingual text efficiently, including Chinese and Korean, across challenging images.Extremely small-scale cases can still expose annotation-related evaluation issues.
- Benchmark comparisons: CTPN achieved the best performance on all five datasets and improved ICDAR 2013 F-measure from 0.80 to 0.88 over recent methods.On ICDAR 2013, precision and recall gains exceeded 5% and 7%, respectively.
5 Conclusions
CTPN is an efficient, end-to-end trainable text detector that localizes text lines from sequential fine-scale proposals in convolutional maps. Its vertical anchors and in-network recurrent connections support accurate localization and challenging-text detection while achieving strong benchmark performance.
- CTPN detects text lines as sequences of fine-scale proposals directly in convolutional feature maps.
- The vertical anchor mechanism jointly predicts each proposal’s precise location and text/non-text score.
- An in-network RNN connects sequential proposals and enables the model to use meaningful image context.
- CTPN detects highly challenging text with fewer false detections and avoids additional post-processing.
- 0.14s/image running time accompanies new state-of-the-art performance on five benchmarks.