Source-linked AI summary
Detecting Oriented Text in Natural Images by Linking Segments
Baoguang Shi, Xiang Bai, Serge Belongie
TL;DR
Text detection methods often target horizontal Latin text and are insufficiently fast for real-time use. SegLink decomposes text into locally detectable oriented segments and connecting links, detects them densely with a fully convolutional network, and combines linked segments into complete text instances. It achieves 75.0% f-measure on ICDAR 2015 Incidental, runs at over 20 FPS on 512x512 images, and detects long non-Latin text without modification.
Problem
Existing text detection methods are often specific to horizontal Latin text and are not fast enough for real-time applications.
Method
SegLink detects oriented segments and links between adjacent segments with a fully convolutional CNN, then combines linked segments into words or text lines.
Results
75.0% f-measure on ICDAR 2015 Incidental, over 20 FPS on 512x512 images, and detection of long non-Latin text without modification.
Takeaways & Limitations
SegLink provides an accurate, efficient, and flexible text detection method for horizontal, oriented, and multilingual text datasets.
Takeaways & Limitations
SegLink requires manual tuning of two thresholds and fails on text with very large character spacing and curved text.
Abstract
from arXiv · showhide
Most state-of-the-art text detection methods are specific to horizontal Latin text and are not fast enough for real-time applications. We introduce Segment Linking (SegLink), an oriented text detection method. The main idea is to decompose text into two locally detectable elements, namely segments and links. A segment is an oriented box covering a part of a word or text line; A link connects two adjacent segments, indicating that they belong to the same word or text line. Both elements are detected densely at multiple scales by an end-to-end trained, fully-convolutional neural network. Final detections are produced by combining segments connected by links. Compared with previous methods, SegLink improves along the dimensions of accuracy, speed, and ease of training. It achieves an f-measure of 75.0% on the standard ICDAR 2015 Incidental (Challenge 4) benchmark, outperforming the previous best by a large margin. It runs at over 20 FPS on 512x512 images. Moreover, without modification, SegLink is able to detect long lines of non-Latin text, such as Chinese.
1. Introduction
SegLink addresses the difficulty of detecting long, oriented, and non-Latin text by decomposing it into locally detectable segments and links, then combining linked segments into words or text lines. Its fully convolutional design detects these elements across scales and achieves strong accuracy, speed, and generality.
- General object detectors are poorly suited to text because text regions have extreme aspect ratios and clear orientations, while some non-Latin text lacks spaces between words.
- SegLink decomposes long text into oriented segments covering parts of words or lines and links connecting adjacent segments from the same text instance.
- Local detection lets SegLink handle text of varying lengths and orientations because segments and links require only local context rather than observing an entire word.
- Segments and links are densely detected by a fully convolutional CNN across six feature layers and combined into whole words by a linking algorithm.
- 75.0% f-measure versus 64.8% for the previous best on ICDAR 2015 Incidental, while processing more than 20 512x512 images per second.
- Without modification, SegLink detects long Latin and non-Latin text lines, including Chinese.
2. Related Work
Prior text detectors use character-, word-, or line-based strategies and often depend on complex extraction or post-processing. SegLink instead jointly predicts segments and links in one end-to-end convolutional pass, using a simpler bottom-up pipeline.
- Character-based methods detect individual characters and group them into words, often using candidate-region extraction or sliding-window classification.
- Line-based methods locate text regions through image segmentation but require sophisticated post-processing for word partitioning or false-positive removal.
- SegLink predicts segments and links jointly in a single forward network pass, producing a simpler, cleaner, and end-to-end trainable pipeline.
- Compared with a CNN plus recurrent neural layers approach, SegLink detects oriented segments with convolutional layers and explicitly detects links for greater flexibility and speed.
- Unlike SSD, which directly regresses bounding boxes, SegLink uses a bottom-up strategy that detects a word or line's component elements and combines them.
3. Segment Linking
SegLink detects oriented text by predicting segments and links across multiple feature layers, then combining linked segments into complete word or text-line boxes. Cross-layer links address redundant multi-scale detections, while graph-based combination produces final bounding boxes.
- Network and output: The feed-forward CNN detects a fixed number of oriented segments and links, filters them by confidence, and combines connected segments into whole word bounding boxes.The network is fully convolutional, and its output count depends on image size.
- Segment detection: Segments are predicted as oriented boxes from default boxes, using a confidence score and five geometric offsets at each feature-map location.The seven segment-prediction channels comprise two score channels and five offset channels.
- Within-layer links: Within-layer links connect each segment to its 8-connected neighbors, indicating shared word membership and helping separate nearby words through negative links.The predictor uses 16 channels for these links, with each pair softmax-normalized into one link score.
- Cross-layer links: Cross-layer links connect segments on adjacent feature layers to resolve redundancies caused by overlapping scale ranges.The first layer has twice the spatial size of the second, yielding four cross-layer neighbors per segment; links are omitted on l = 1.
- Cross-layer links: Cross-layer linking provides a trainable alternative to traditional non-maximum suppression for joining detections of different scales.The approach fits the existing linking strategy and is easy to implement within the framework.
- Combining segments: Final combination treats filtered segments as graph nodes and filtered links as edges, finds connected components with DFS, and constructs one combined bounding box per component.For each component, segment centers are fitted to a straight line and projected points determine the box extent.
4. Training
Training derives supervision for segments and links from groundtruth word bounding boxes, then jointly optimizes segment classification, geometry regression, and link classification.
- Groundtruth construction: Groundtruth supervision includes default-box labels, offsets to matched segments, and labels for within-layer and cross-layer links.These targets are calculated from groundtruth word bounding boxes.
- Groundtruth construction: Positive default boxes must have centers inside a word bounding box and satisfy a size-to-height criterion; others are negative.With multiple words, qualifying boxes are matched to the word with the closest size.
- Groundtruth construction: Groundtruth segments are calculated for positive default boxes, whose offsets are then obtained by solving Eq. 2 to Eq. 6.The procedure follows the steps illustrated in Figure 5.
- Groundtruth construction: A link is positive only when both connected default boxes are positive and matched to the same word.
- Optimization: The network jointly minimizes segment classification, offset regression, and link classification losses through a weighted-sum objective.Segment labels distinguish positive default boxes, while link labels identify positive links; classification uses softmax loss and geometry uses Smooth L1 regression.
- Optimization: Hard negative mining keeps the negative-to-positive ratio at most 3:1 and is performed separately for segments and links.
5. Experiments
Experiments on IC15, TD500, and IC13 evaluate SegLink across incidental, multilingual oriented, and mostly horizontal text settings. The method achieves strong accuracy and speed, while failures remain for large character spacing and curved text.
- Datasets: SegLink is evaluated on ICDAR 2015 Incidental Text, MSRA-TD500, and ICDAR 2013 using each dataset’s standard evaluation protocol.IC15 contains incidental text with varied orientation, scale, and resolution; TD500 contains multilingual oriented text; IC13 is mostly horizontal.
- Detecting Oriented English Text: SegLink distinguishes text from cluttered backgrounds and separates nearby words through explicit link prediction.The links are predicted as negative between nearby words that should remain separate.
- Detecting Multi-Lingual Text in Long Lines: SegLink achieves the highest precision and f-measure on TD500 and runs at 8.9 FPS.The experiment combines TD500 and IC15 training data because TD500 has only 300 training images.
- Detecting Multi-Lingual Text in Long Lines: On TD500, densely detected segments and links form long bounding boxes for mixed English and Chinese text without structural modification.The segment-link representation handles long lines that are difficult for conventional object detectors to obtain directly.
- Detecting Horizontal Text: On IC13, SegLink achieves competitive f-measure and exceeds 20 FPS on 512 × 512 images, while only one approach has higher f-measure.The comparison uses the Deteval protocol except for methods marked as evaluated under the ICDAR 2013 protocol.
- Limitations: SegLink fails on text with very large character spacing and curved text; curved-text failure is attributed to a rectangle-only segment combination algorithm.For large spacing, adjacent segments are detected but distant segments are not linked.
6. Conclusion
SegLink is a simple, highly efficient CNN-based text detection strategy that performs accurately across horizontal, oriented, and multilingual text datasets while remaining fast and flexible.
- SegLink is implemented as a simple, highly efficient CNN model.
- SegLink performs well on horizontal, oriented, and multilingual text datasets.
- The authors conclude that SegLink is accurate, fast, and flexible.