Source-linked AI summary
Towards End-to-end Text Spotting with Convolutional Recurrent Neural Networks
Hui Li, Peng Wang, Chunhua Shen
TL;DR
Text spotting in natural scenes requires both localization and recognition, yet prior systems commonly handle these tasks separately. This paper introduces an end-to-end convolutional recurrent network that performs both tasks in one forward pass, with shared features and aspect-ratio-aware region encoding. The method reports competitive benchmark performance, including higher detection F-measures than its detection-only counterpart and improved recall over TextBoxes in a stated comparison.
Problem
Natural-scene text spotting remains challenging, while existing systems commonly separate detection and recognition despite their correlated information.
Method
An end-to-end network jointly detects and recognizes words using shared convolutional features, an aspect-ratio-preserving region encoder, and recurrent recognition components.
Results
The best model reportedly outperforms state-of-the-art results on several benchmarks; its detection F-measures are 85.6% on ICDAR2015 and 85.1% on ICDAR2011, averaging 2% above its detection-only counterpart.
Takeaways & Limitations
Detection and recognition can be integrated into a single end-to-end forward pass while sharing convolutional computation and avoiding intermediate character grouping or text-line separation.
Abstract
from arXiv · showhide
In this work, we jointly address the problem of text detection and recognition in natural scene images based on convolutional recurrent neural networks. We propose a unified network that simultaneously localizes and recognizes text with a single forward pass, avoiding intermediate processes like image cropping and feature re-calculation, word separation, or character grouping. In contrast to existing approaches that consider text detection and recognition as two distinct tasks and tackle them one by one, the proposed framework settles these two tasks concurrently. The whole framework can be trained end-to-end, requiring only images, the ground-truth bounding boxes and text labels. Through end-to-end training, the learned features can be more informative, which improves the overall performance. The convolutional features are calculated only once and shared by both detection and recognition, which saves processing time. Our proposed method has achieved competitive performance on several benchmark datasets.
1. Introduction
Scene text spotting remains difficult because natural images contain highly variable text patterns and complicated backgrounds. The paper proposes jointly detecting and recognizing words in one end-to-end trainable network, sharing features and avoiding intermediate processing.
- Natural scene text spotting remains an open problem because text patterns vary widely and backgrounds are highly complicated.
- Previous systems commonly detect text first, then recognize words from cropped regions using separate processing stages.
- The proposed network jointly detects and recognizes words in one forward evaluation, avoiding character grouping and text-line separation.
- Shared convolutional features support both detection and recognition, while end-to-end training is designed to improve overall performance and save processing time.
- The region feature extractor preserves varying text aspect ratios and uses an RNN encoder to produce fixed-length representations.
- Curriculum learning progresses from synthetic images with large lexicons to real images with small lexicons; the best model reportedly outperforms state-of-the-art methods on several benchmarks.
2. Related Work
Related text spotting systems typically divide detection and recognition into separate stages, using character-, text-line-, or word-based detection and diverse recognition strategies. The paper’s approach is positioned against these pipelines by targeting simultaneous detection and recognition.
- Text spotting combines text detection, which localizes words, and word recognition, which identifies their content.
- Text Detection: Character-based detectors find individual characters and group them into words using sliding windows or connected components.
- Text Detection: Text-line-based methods detect lines first and subsequently separate them into individual words.
- Text Detection: Word-based DNN detectors directly localize words using architectures including Faster R-CNN, YOLO, SSD, and related proposal networks.
- Word Recognition: Recognition methods range from bottom-up character integration and large-dictionary classification to RNN sequence labeling and attention-based sequence-to-sequence models.
- Text Spotting Systems: Earlier text spotting systems often generate high-recall proposals and refine them with separate recognition models or heuristic grouping procedures.
3. Model
The model is an end-to-end architecture that jointly proposes, detects, and recognizes text regions while sharing convolutional features. It preserves text-region aspect ratios during encoding and uses recurrent networks for fixed-length representations and attention-based recognition.
- Overall Architecture: The system feeds an image through convolutional layers, then applies TPN, RFE, TDN, a second RFE, and TRN for detection and recognition.Convolutional features are computed once and reused by later stages.
- Text Proposal Network: TPN uses multi-scale sliding windows, local and contextual features, textness classification, and coordinate regression to generate text proposals.Its anchors cover four scales and six aspect ratios designed for variable-sized, wide word boxes.
- Region Feature Encoder: RFE preserves each region’s aspect ratio by producing variable-length pooled feature maps, then uses an LSTM to encode them into fixed-dimensional vectors.The final hidden state hW has dimension 1024 and represents the entire region.
- Text Detection Network: TDN classifies proposal regions as text or non-text and refines their bounding-box coordinates using fully connected layers followed by parallel classification and regression layers.The system refines bounding boxes first in TPN and again in TDN.
- Text Recognition Network: TRN encodes region features with LSTMs and decodes words through an attention-based sequence-to-sequence model using encoder hidden states as context.During training, decoder inputs include ground-truth word tokens and attention outputs guided by the previous decoder state.
4. Experiments
Experiments evaluate the unified text spotter across benchmarks, model settings, training strategies, and comparisons with prior systems. The attention and varying-size RoI design improve recognition, while joint training improves detection and the full model achieves strong efficiency and benchmark performance.
- Model settings: Attention improves F-measures across all evaluated data compared with the no-attention fixed-pooling model.The attention decoder selects local features corresponding to each character during decoding.
- Model settings: Varying-size RoI pooling increases F-measures by around 1% on ICDAR2015, 4% on ICDAR2011, and 3% on SVT with the Strong lexicon.It is especially beneficial for long words, where fixed-size pooling can discard substantial information.
- Joint versus separate training: Joint training raises detection F-measures to 85.6% on ICDAR2015 and 85.1% on ICDAR2011, averaging about 2% above detection-only training.The jointly trained and detection-only systems share architecture but use different training objectives.
- Comparison with other methods: The proposed model outperforms compared methods on most evaluated datasets and yields 1.5% higher average recall than TextBoxes with a generic lexicon.It uses three input scales, compared with five for TextBoxes.
- Qualitative results and efficiency: The model handles words with varied aspect ratios and orientations, and processes a 600 × 800 image in approximately 0.9s on an M40 GPU.Shared convolutional features avoid recomputation during recognition in the unified system.
5. Conclusion
The paper presents a unified end-to-end DNN that simultaneously detects and recognizes scene text in one forward pass. Its varying-aspect-ratio RoI encoding supports efficient and accurate benchmark performance, while multi-oriented text remains a future direction.
- Conclusion: The proposed DNN simultaneously detects and recognizes text in natural scene images through end-to-end training and a single forward pass.The framework is designed for efficient and accurate scene text spotting.
- Conclusion: The novel RoI encoding method accounts for the large diversity of word bounding-box aspect ratios.The paper identifies handling images with multi-oriented text as potential future work.
6. Appendix
The appendix describes training data that progresses from simple synthetic images to complex synthetic scenes and finally real-world images. It also compares varying-size pooling with fixed-size pooling and presents attention and text-spotting results.
- 6.1. Training Data with Different Levels of Complexity: The Synth800k dataset adds 800k images blending rendered words into natural scenes with more complex backgrounds.This stage further tunes the model for complicated appearance patterns.
- 6.1. Training Data with Different Levels of Complexity: Finally, 2044 naturally captured images focused on text are used to fine-tune the model.Figure 9 shows examples from ICDAR2015, SVT, and AddF2k.
- Experimental Results: Figure 10 visualizes attention weights during RNN encoding, while Figure 11 presents text-spotting results from the proposed Ours Atten+Vary model.The result examples cover multiple datasets and the attention-based decoding variants include Ours Atten+Vary and Ours Atten+Fixed.