Source-linked AI summary
Towards Unconstrained End-to-End Text Spotting
Siyang Qin, Alessandro Bissacco, Michalis Raptis, Yasuhisa Fujii, Ying Xiao
TL;DR
Reading arbitrary-shaped scene text remains difficult because existing OCR pipelines and end-to-end methods struggle with irregular geometry. The paper combines instance segmentation, unrectified attention decoding, RoI masking, and partially labeled training data, surpassing prior results on both straight and curved benchmarks.
Problem
Existing OCR pipelines accumulate errors and dependencies, while end-to-end methods struggle to generalize from straight text to curved text.
Method
The model uses Mask R-CNN instance segmentation, an attention decoder operating on RoI-masked unrectified features, and partially labeled data from an existing OCR engine.
Results
The method surpasses prior state of the art by 4.6% on ICDAR15 and by more than 16% on Total-Text.
Takeaways & Limitations
End-to-end training can detect and recognize arbitrary-shaped text while achieving strong performance on both straight and curved OCR benchmarks.
Abstract
from arXiv · showhide
We propose an end-to-end trainable network that can simultaneously detect and recognize text of arbitrary shape, making substantial progress on the open problem of reading scene text of irregular shape. We formulate arbitrary shape text detection as an instance segmentation problem; an attention model is then used to decode the textual content of each irregularly shaped text region without rectification. To extract useful irregularly shaped text instance features from image scale features, we propose a simple yet effective RoI masking step. Additionally, we show that predictions from an existing multi-step OCR engine can be leveraged as partially labeled training data, which leads to significant improvements in both the detection and recognition accuracy of our model. Our method surpasses the state-of-the-art for end-to-end recognition tasks on the ICDAR15 (straight) benchmark by 4.6%, and on the Total-Text (curved) benchmark by more than 16%.
1. Introduction
The paper addresses the difficulty of reading arbitrary-shaped scene text by replacing cascaded OCR with an end-to-end model that detects and recognizes text in one pass. It combines Mask R-CNN, attention decoding, RoI masking, and partially labeled data to improve performance on straight and curved benchmarks.
- Motivation: End-to-end OCR avoids cascade errors and dependencies that make joint optimization, adaptation, and maintenance difficult in traditional pipelines.Prior end-to-end methods nevertheless struggle with curved text, making arbitrary-shaped text a central open problem.
- Results: The model surpasses prior state of the art by 4.6% end-to-end F-score on ICDAR15 and by more than 16% on Total-Text.These benchmarks cover straight and curved text, respectively.
- Method: The proposed end-to-end OCR model combines a Mask R-CNN detector with an attention decoder to recognize arbitrary-shaped text.It predicts text locations and transcriptions in a single forward pass.
- Method: RoI masking removes the need for feature rectification, allowing the attention decoder to operate directly on irregular text instances.The decoder can follow text in different orientations and along curved paths.
- Training: Partially labeled data bootstrapped from an existing multi-step OCR engine addresses the shortage of fully annotated images for joint training.The recognizer requires more data and iterations than the detector, creating a risk of detector overfitting during joint training.
2. Related Work
Related work has advanced curved-text detection and end-to-end OCR, but recognition commonly assumes rectified text. The paper instead uses instance masks and unrectified features to handle arbitrary shapes.
- Scene Text Detection: Curved-text detection has advanced through polygon boxes and instance representations, but reading curved text remains unsolved.The related datasets provide tight polygon boxes and ground-truth transcriptions.
- Scene Text Recognition: Most scene-text recognizers assume rectified, straight, left-to-right inputs and use CTC- or attention-based architectures.The input is typically resized to a constant height before feature extraction.
- End-to-End OCR: Prior end-to-end systems use rotated rectangles or quadrilaterals to compute rectified text features, whereas this method uses rectangular boxes and instance masks.Removing rectification lets the attention decoder operate on cropped and masked features for curved text.
- End-to-End OCR: A prior Mask R-CNN OCR system detects individual characters, which can lose sequential information and make character linking difficult.The paper instead decodes textual content with an attention model.
3. Model Architecture and Training
The model combines Mask R-CNN instance segmentation with an attention-based seq2seq recognizer that reads arbitrarily shaped text without feature rectification. RoI masking, multi-scale features, and partially labeled data support recognition and joint training across straight and curved text.
- Detector: Mask R-CNN predicts each text instance’s axis-aligned box and segmentation mask, enabling the model to handle both straight and curved text paths.Straight detections use fitted min-area rotated rectangles, while curved detections use fitted polygons.
- RoI Masking: RoI masking multiplies cropped image features by the instance mask, allowing the recognizer to process arbitrarily shaped text without rectification.This removes neighboring text and preserves the irregular text shape for recognition.
- Multi-Scale Feature Fusion: The backbone uses output-stride-8 features and fused higher-resolution features to provide dense representations for fine-grained text recognition.ResNet-50 and Inception-ResNet backbones use atrous convolutions to maintain receptive field while reducing stride.
- Recognizer: The recognizer uses a Bahdanau-style seq2seq attention decoder that predicts symbols sequentially from masked text features.At each step, the LSTM uses the previous symbol, previous state, and an attention-weighted image context; attention can follow arbitrary text paths.
- Joint Training and Loss Function: Partially labeled images from an existing OCR engine train only the recognizer, while fully labeled images jointly train detector and recognizer branches.This supplies additional recognition data while preventing detector overfitting; the model uses a single-step joint-training strategy.
- Implementation Details: The training data combines public datasets with 30k manually labeled web images containing oriented boxes and transcriptions.Public sources include SynthText, ICDAR15, COCO-Text, ICDAR-MLT, and Total-Text.
4. Experiments
On ICDAR15 and Total-Text, the model achieves strong detection and end-to-end recognition, including substantial gains on curved text, while ablations clarify the value of RoI masking and single-step training.
- 4.1. Straight Text: 4.6% higher end-to-end F-score is achieved than multi-scale FOTS on ICDAR15 while matching its detection performance.The method also exceeds the best single-scale detection model by 1.8% and the highest single-scale end-to-end model by about 7%.
- 4.2. Curved Text: 16.7% higher end-to-end recognition performance is achieved on Total-Text than the previous state of the art.The best model also improves detection by 5.1%; using only public datasets still yields gains of 4.2% in detection and 9.9% in recognition.
- 4.2. Curved Text: The model produces high-quality polygons and transcriptions, handles some partial occlusion, and fails mainly on rare upside-down right-to-left text.Attention visualizations show the decoder focusing on the relevant area for each symbol and following curved text paths.
- 4.3. Ablation Experiments: RoI masking consistently improves end-to-end AP, including a +3.3% gain with ResNet-50 and partially labeled data.Detection effects are mixed, with marginal improvement for Inception-ResNet but degradation for ResNet-50.
- 4.3. Ablation Experiments: Single-step joint training consistently outperforms two-step training for both detection and end-to-end evaluation.It jointly optimizes detector and recognizer using fully and partially labeled data and is simpler for hyperparameter tuning and neural architecture search.
- 4.3. Ablation Experiments: End-to-end inference takes 210 ms with ResNet-50 and 330 ms with Inception-ResNet on 1280×720 ICDAR15 images.The recognition branch adds relatively little overhead compared with detection-only inference times of 180 ms and 270 ms, respectively.
5. Conclusion
The paper presents an end-to-end network that detects and recognizes text of arbitrary shape using Mask R-CNN, attention decoding, RoI masking, and partially machine-annotated data. It substantially surpasses prior methods on ICDAR15 and Total-Text while remaining reasonably efficient.
- The network simultaneously detects and recognizes arbitrary-shaped text using Mask R-CNN, an attention decoder, RoI masking, and partially machine-annotated data.