Source-linked AI summary
An end-to-end TextSpotter with Explicit Alignment and Attention
Tong He, Zhi Tian, Weilin Huang, Chunhua Shen, Yu Qiao, Changming Sun
TL;DR
Text spotting is difficult to train jointly because detection and recognition differ substantially in optimization difficulty, while sequential pipelines accumulate errors and unified systems have alignment or feature-sharing limitations. The paper presents an end-to-end single-shot textspotter combining orientation-aware text alignment, explicitly supervised character attention, and a shared RNN recognition branch. On ICDAR2015, F-measure improved from (0.54, 0.51, 0.47) to (0.82, 0.77, 0.63) across strong, weak, and generic lexicons, and joint training also improved detection.
Problem
Text spotting must handle diverse, complicated natural-image text, but cascaded detection and recognition accumulate errors while joint training is difficult because the tasks have different optimization difficulties.
Method
The model jointly detects and recognizes text using a text-alignment layer, explicitly supervised character attention, and a parallel RNN branch sharing convolutional features end to end.
Results
F-measure on ICDAR2015 improved from (0.54, 0.51, 0.47) to (0.82, 0.77, 0.63) with strong, weak, and generic lexicons, respectively.
Takeaways & Limitations
Joint detection and recognition are complementary in this model: recognition improves detection accuracy, while shared training supports challenging text instances.
Abstract
from arXiv · showhide
Text detection and recognition in natural images have long been considered as two separate tasks that are processed sequentially. Training of two tasks in a unified framework is non-trivial due to significant dif- ferences in optimisation difficulties. In this work, we present a conceptually simple yet efficient framework that simultaneously processes the two tasks in one shot. Our main contributions are three-fold: 1) we propose a novel text-alignment layer that allows it to precisely compute convolutional features of a text instance in ar- bitrary orientation, which is the key to boost the per- formance; 2) a character attention mechanism is introduced by using character spatial information as explicit supervision, leading to large improvements in recognition; 3) two technologies, together with a new RNN branch for word recognition, are integrated seamlessly into a single model which is end-to-end trainable. This allows the two tasks to work collaboratively by shar- ing convolutional features, which is critical to identify challenging text instances. Our model achieves impressive results in end-to-end recognition on the ICDAR2015 dataset, significantly advancing most recent results, with improvements of F-measure from (0.54, 0.51, 0.47) to (0.82, 0.77, 0.63), by using a strong, weak and generic lexicon respectively. Thanks to joint training, our method can also serve as a good detec- tor by achieving a new state-of-the-art detection performance on two datasets.
1. Introduction
Text spotting remains difficult because natural-image text varies widely in font, scale, orientation, illumination, and background complexity. The paper introduces an end-to-end single-shot model that jointly detects and recognizes text using aligned features, character attention, and shared convolutional representations.
- Motivation: Text spotting maps a natural image to text transcripts and their locations, but diverse appearances and complicated backgrounds keep it an open problem.The challenge includes variation in font, scale, orientation, illumination, and background.
- Method: The proposed single-shot textspotter jointly processes detection and word recognition through parallel branches sharing convolutional features in an end-to-end trainable framework.A new RNN recognition branch operates alongside the detection branch.
- Method: The text-alignment layer uses grid sampling to produce fixed-length features precisely aligned with arbitrarily oriented detected text regions, avoiding RoI-pooling misalignment.This preserves spatial information while avoiding orientation-changing and quantization effects.
- Method: Character attention uses explicit character spatial supervision to guide RNN decoding toward relevant features, improving word recognition.The mechanism encodes character locations and attentions into the recognition branch.
- Results: F-measure on ICDAR2015 improved from (0.54, 0.51, 0.47) to (0.82, 0.77, 0.63) with strong, weak, and generic lexicons, respectively.Experiments also found that recognition significantly improves detection accuracy.
- Limitations of prior work: Prior systems commonly cascade detection and recognition, accumulating training errors, while existing unified approaches face orientation or feature-sharing limitations.One unified method is limited to horizontal examples, whereas another lacks shared features linking recognition loss to localization.
2. Single Shot TextSpotter by Joint Detection and Recognition
The textspotter jointly detects text and recognizes words in one shot, using aligned text-region features and character-guided attention within a parallel recognition branch. Its training strategy addresses unequal task difficulty and missing character-level annotations.
- Joint TextSpotter: The model directly maps an image to word transcripts and arbitrarily oriented bounding boxes through parallel detection and recognition branches.It produces final results in one shot, requiring only simple non-maximum suppression as post-processing.
- Joint TextSpotter: Feature fusion combines convolutional maps across layers to preserve local detail and high-level context for multi-scale text instances.The architecture uses conv5, conv4, conv3, and conv2 features with a PVA backbone.
- Text-Alignment Layer: Text-alignment sampling computes fixed-size features precisely inside arbitrarily oriented quadrilateral regions, avoiding RoI-pooling quantization and misalignment.Grid sampling and bilinear interpolation provide per-pixel alignment while reducing irrelevant text and background encoding.
- Attention Alignment and Enhancement: Character attention uses explicit spatial supervision to guide decoding toward character features and improve recognition alignment.The recognition branch uses character information and an additional alignment loss, while position embedding adds attention-location information.
- Training: End-to-end joint training is difficult because recognition is harder to optimize and character-level annotations are absent from standard benchmarks.A multi-step training strategy enables collaborative training and generalization from synthesized images to real-world data.
3. Experiments
Experiments evaluate the unified textspotter on ICDAR2013 and ICDAR2015 using multiple protocols, lexicons, ablations, and comparisons with prior methods. The results show strong recognition, detection, and joint-training performance, including gains from text alignment and character attention.
- Evaluation Protocols: The evaluation covers ICDAR2013, ICDAR2015, and a multilingual scene-text dataset, using detection and end-to-end recognition protocols.ICDAR2013 emphasizes horizontal text, whereas ICDAR2015 includes varied scale, blur, and orientation; the multilingual dataset contributes Latin-script training data.
- Text-alignment vs. RoI Pooling: 67.6% recognition accuracy is achieved with the proposed text-alignment layer, compared with 60.7% using standard RoI pooling.The comparison fixes detection to ground truth and evaluates recognition on multi-orientation ICDAR2015 text without a lexicon at single scale.
- Character Attention: 0.95 and 0.88 accuracy are obtained with character supervision, versus 0.93 and 0.85 for traditional attention on synthetic data.On ICDAR2015, the proposed character localization also yields about 2% higher recognition accuracy than attentional LSTM.
- Joint Training vs. Separate Models: Joint training improves detection F-measure by 3% and makes detection more robust to text-like backgrounds and complicated text instances.The detection results are reported without using recognition outputs, while the comparison concerns joint versus separate training.