Source-linked AI summary
FOTS: Fast Oriented Text Spotting with a Unified Network
Xuebo Liu, Ding Liang, Shi Yan, Dagui Chen, Yu Qiao, Junjie Yan
TL;DR
Scene text spotting is difficult, and conventional systems separate detection from recognition, increasing cost and missing shared visual supervision. FOTS unifies both tasks end to end with shared convolutions and RoIRotate. Across standard benchmarks, it reports stronger efficiency and performance, including real-time operation.
Problem
Separate detection and recognition systems incur heavy time costs and ignore correlations in visual cues shared by the two tasks.
Method
FOTS is an end-to-end trainable oriented text spotting framework that shares convolutional features and uses RoIRotate to connect detection with recognition.
Results
FOTS significantly outperforms previous methods on standard benchmarks in efficiency and performance while maintaining real-time speed.
Takeaways & Limitations
Shared convolutional features make the text recognition step nearly cost-free, enabling real-time oriented scene text spotting.
Takeaways & Limitations
During training, FOTS uses ground-truth text regions instead of predicted regions because recognition is sensitive to detection noise.
Abstract
from arXiv · showhide
Incidental scene text spotting is considered one of the most difficult and valuable challenges in the document analysis community. Most existing methods treat text detection and recognition as separate tasks. In this work, we propose a unified end-to-end trainable Fast Oriented Text Spotting (FOTS) network for simultaneous detection and recognition, sharing computation and visual information among the two complementary tasks. Specially, RoIRotate is introduced to share convolutional features between detection and recognition. Benefiting from convolution sharing strategy, our FOTS has little computation overhead compared to baseline text detection network, and the joint training method learns more generic features to make our method perform better than these two-stage methods. Experiments on ICDAR 2015, ICDAR 2017 MLT, and ICDAR 2013 datasets demonstrate that the proposed method outperforms state-of-the-art methods significantly, which further allows us to develop the first real-time oriented text spotting system which surpasses all previous state-of-the-art results by more than 5% on ICDAR 2015 text spotting task while keeping 22.6 fps.
1. Introduction
FOTS unifies oriented text detection and recognition in one end-to-end trainable network, addressing the cost and missed feature-sharing opportunities of two-stage systems. RoIRotate connects oriented detection regions to recognition while shared convolutions support real-time operation and strong benchmark performance.
- Scene text reading remains difficult because text patterns vary widely and backgrounds are highly complicated.
- Separate detection and recognition pipelines incur heavy time costs and fail to exploit correlations in their shared visual cues.
- FOTS jointly detects and recognizes text through shared convolutional features and complementary supervision in an end-to-end trainable framework.
- The architecture predicts text regions, extracts proposal features, and recognizes labels through an RNN encoder and CTC decoder in one differentiable system.
- RoIRotate extracts oriented text-region features from convolutional maps using detected oriented bounding boxes, connecting detection with recognition.
- FOTS significantly surpasses state-of-the-art methods on ICDAR 2015, ICDAR 2017 MLT, and ICDAR 2013 benchmarks.
2. Related Work
Prior work includes character-based and direct word-detection methods, diverse recognition architectures, and predominantly two-stage spotting pipelines. The supplied passages also identify an earlier end-to-end method designed only for horizontal text, contrasting with FOTS's oriented-text focus.
- Conventional text detection localizes characters and groups them into words or text lines using sliding-window or connected-components methods.
- Deep detection methods directly predict words using vertical anchors, rotation-based proposals, text-segment linkage, or dense direct regression.
- Scene text recognition methods include word classification, sequence labelling with CTC, and sequence-to-sequence models.
- Most previous text spotting systems first generate text proposals with detection models and then recognize them using separate recognition models.
- An earlier end-to-end text spotting method combined a proposal network with LSTM attention recognition but was suitable only for horizontal text.
3. Methodology
FOTS combines oriented text detection and recognition in one end-to-end network through shared convolutional features and the differentiable RoIRotate operator. The architecture preserves text-region geometry for recognition while using task-specific losses and training choices for efficient spotting.
- Overall Architecture: FOTS simultaneously detects and recognizes words using shared convolutions, a detection branch, RoIRotate, and a recognition branch.The recognition branch follows RoIRotate with CNN/LSTM sequence encoding and CTC decoding.
- Text Detection Branch: The detection branch uses a fully convolutional network to produce dense per-pixel word predictions from upscaled shared feature maps.Shared features are maintained at 1/4 of the input resolution to support small text boxes.
- Training: Online hard example mining improves discrimination of text-like patterns and increases ICDAR 2015 F-measure by about 2%.It also addresses class imbalance during detection training.
- RoIRotate: RoIRotate transforms oriented feature regions into axis-aligned representations with fixed height and unchanged aspect ratio for recognition.Unlike fixed-size pooling, the operation preserves proposal width variation and uses bilinear interpolation.
- RoIRotate: RoIRotate applies affine transformations to shared feature maps, using proposal coordinates and orientation from ground truth or the detection branch.The transformed outputs are canonical horizontal feature maps for each text region.
- Text Recognition Branch: The recognition branch reduces features only twice along width, then applies height-wise sequential convolutions and pooling before sequence recognition.This preserves discriminable features for narrow characters while extracting higher-level spatial features.
4. Experiments
Experiments evaluate FOTS on three public benchmarks and compare joint training with two-stage and state-of-the-art systems. FOTS improves detection and spotting while maintaining near-real-time speed through shared convolutional features.
- Benchmark datasets: FOTS is evaluated on ICDAR 2015, ICDAR 2017 MLT, and ICDAR 2013 for text localization and text spotting.ICDAR 2017 MLT has no text spotting task, so only text detection is reported there.
- Comparison with two-stage method: Jointly trained FOTS significantly outperforms “Our Detection” in text localization and “Our Two-Stage” in text spotting.The comparison is reported across Tables 2, 3, and 4.
- Comparison with two-stage method: FOTS reduces Miss, False, Split, and Merge detection errors compared with “Our Detection” by using character-level information from recognition supervision.Recognition supervision helps distinguish characters from similar backgrounds and preserve continuous text regions.
- Comparisons with state-of-the-art results: On ICDAR 2015 text spotting, FOTS exceeds the previous best method by more than 15% in F-measure.ICDAR 2013 contains horizontal text, while ICDAR 2015 and ICDAR 2017 MLT include arbitrarily oriented text regions.
5. Conclusion
FOTS is an end-to-end framework for oriented scene text spotting that unifies detection and recognition with RoIRotate. Shared convolutional features make recognition nearly cost-free, enabling real-time operation while improving benchmark efficiency and performance.
- 5. Conclusion: FOTS is an end-to-end trainable framework for oriented scene text spotting.The framework jointly handles text detection and recognition.
- 5. Conclusion: RoIRotate unifies text detection and recognition into an end-to-end pipeline by extracting oriented text regions from convolutional feature maps.The operation connects the detection and recognition stages.
- 5. Conclusion: Shared convolutional features make the text recognition step nearly cost-free and enable real-time speed.The conclusion attributes the efficiency to sharing convolutional features.
- 5. Conclusion: Experiments on standard benchmarks show that FOTS significantly outperforms previous methods in efficiency and performance.