Source-linked AI summary
Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition
Hui Li, Peng Wang, Chunhua Shen, Guyu Zhang
TL;DR
Irregular scene text recognition must handle curvature, orientation, and distortion without relying on cumbersome designs or fine-grained annotations. The paper proposes SAR, combining a ResNet, LSTM encoder-decoder, and weakly supervised 2D attention; it reports strong performance on both irregular and regular text.
Problem
Irregular scene text varies in curvature, orientation, and distortion, while existing recognizers often depend on sophisticated designs or extra fine-grained annotations.
Method
SAR uses a 31-layer ResNet, an LSTM-based encoder-decoder, and tailored 2D attention trained with only word-level annotations.
Results
SAR outperforms previous methods on irregular text datasets and achieves comparable results on regular text.
Takeaways & Limitations
A simple architecture built from off-the-shelf components provides a strong baseline for recognizing irregular natural-scene text.
Takeaways & Limitations
Reported failure cases include blurry images, partial occlusion, extreme distortion, uneven lighting, uncommon fonts, and vertical text.
Abstract
from arXiv · showhide
Recognizing irregular text in natural scene images is challenging due to the large variance in text appearance, such as curvature, orientation and distortion. Most existing approaches rely heavily on sophisticated model designs and/or extra fine-grained annotations, which, to some extent, increase the difficulty in algorithm implementation and data collection. In this work, we propose an easy-to-implement strong baseline for irregular scene text recognition, using off-the-shelf neural network components and only word-level annotations. It is composed of a $31$-layer ResNet, an LSTM-based encoder-decoder framework and a 2-dimensional attention module. Despite its simplicity, the proposed method is robust and achieves state-of-the-art performance on both regular and irregular scene text recognition benchmarks. Code is available at: https://tinyurl.com/ShowAttendRead
Introduction
Irregular scene text recognition remains difficult because text can be curved, oriented, or distorted, while existing methods often require sophisticated designs or fine-grained annotations. SAR addresses this with standard components and weakly supervised 2D attention, achieving strong performance on irregular and regular text.
- 76.1% accuracy is reported for the top-performing existing approach on ICDAR 2015 irregular text recognition.
- SAR combines a 31-layer ResNet, an LSTM encoder-decoder, and tailored 2D attention for irregular text recognition.
- SAR outperforms previous methods on irregular text datasets while achieving comparable results on regular text.
- The model uses standard neural modules, can be implemented in around 100 lines of code, and trains without pre-training using synthetic or public real data.
- Word-level annotations suffice because the attention module selects local character features under cross-entropy supervision, without pixel- or character-level labels.
Related Work
Prior scene text recognition methods include bottom-up character integration, rectification-based approaches, and attention or hierarchical architectures. These methods face challenges from difficult character separation, severe distortions, or additional annotation and architectural requirements.
- Early Work: Early methods detect individual characters and integrate them into words using dynamic programming or graph models.
- Early Work: Character detection or separation remains difficult because of complicated backgrounds and cursive fonts.
- Rectification-based Methods: Rectification methods remove distortions before recognition, while Char-Net rectifies individual characters but requires extra character-level annotations.
- Attention-based Methods: Hierarchical attention and multi-directional encoding are examples of sophisticated designs used for irregular text recognition.
Model
The model extracts a variable-width 2D feature map with a 31-layer ResNet, then processes its columns with an LSTM encoder. The resulting representation supports recognition across varying image aspect ratios.
- Architecture: The framework uses a ResNet CNN followed by a 2D-attention encoder-decoder that outputs a variable-length character sequence.
- ResNet CNN: The 31-layer ResNet uses convolutional and max-pooling layers to produce 2D feature maps for holistic representation and attention context.
- ResNet CNN: A 1 × 2 max-pooling layer preserves more horizontal information and benefits recognition of narrow characters such as ‘i’ and ‘l’.
- ResNet CNN: Input images are resized to fixed height and varying width, so the feature-map width varies with image aspect ratio.
- LSTM Encoder: The LSTM encoder max-pools each feature-map column vertically before feeding it into the recurrent sequence.
2D Attention based Encoder-Decoder
The model uses a 2D attention-based encoder-decoder to recognize irregular text without transforming the original image. An LSTM encoder produces a holistic feature, while the decoder uses attention-derived glimpses to generate characters.
- Encoder: The encoder processes columns of 2D feature maps with a two-layer LSTM and uses its final hidden state h_W as the image’s holistic feature.The encoder applies vertical max-pooling before updating its hidden state.
- Decoder: The decoder receives the holistic feature, a START token, and previous outputs before terminating generation with an END token.At each step, the output is computed from the current hidden state and the attention output.
- 2D Attention: The attention weight for each feature location incorporates neighborhood information through learned transformations and convolution-compatible operations.The resulting glimpse is a weighted sum of local features.
Experiments
The experiments evaluate the proposed method across public benchmarks containing both regular and irregular natural-scene text, with ablations examining model hyperparameters and training data.
- Evaluation: The evaluation compares the model with state-of-the-art methods on multiple public benchmarks covering regular and irregular scene text.The section also reports ablation studies of model hyperparameters and training data.
Datasets
The experiments use large synthetic datasets for training and several real-image benchmarks for testing, spanning horizontal, perspective-distorted, and curved text.
- Synthetic Datasets: Syn90k and SynthText provide large synthetic word-recognition datasets, while SynthAdd supplements them with additional special-character examples.SynthAdd contains 1.6 million synthesized word images with randomly inserted special characters.
- Attention Implementation: Figure 5 illustrates that the proposed attention computation can be implemented with convolutions, including a 3 × 3 convolution for neighborhood aggregation.The tile operation duplicates a 1 × 1 × d vector across an H × W feature map.
- Real Datasets: IIIT5K contains 5,000 nearly horizontal word patches, with 2,000 for training and 3,000 for testing.Each image has both a 50-word and a 1,000-word lexicon.
- Real Datasets: SVT, SVTP, and CUTE80 test recognition on nearly horizontal, perspective-distorted, and curved text images, respectively.SVT has 647 test patches, SVTP has 639, and CUTE80 has 288 curved-text images.
Implementation Details
The model is trained end-to-end with cross-entropy loss and ADAM, using mixed synthetic and real training data. At inference, rotated views and beam search support recognition.
- Training: The network is trained end-to-end without pre-training using cross-entropy loss, ADAM, and batches of 32.The learning rate starts at 10^-3 and decays by 0.9 every 10,000 iterations until reaching 10^-5.
- Training Data: Training groups combine 120k Syn90k patches, 120k SynthText patches, 80k SynthAdd patches, and approximately 50k real-training patches.The data groups are constructed iteratively.
- Inference: For tall test images, the system recognizes clockwise, anticlockwise, and original orientations, then selects the result with the highest averaged output probability.This provides an inference-time procedure for handling rotated text images.
- Inference: LSTM decoding uses beam search with k empirically set to 5, retaining candidates with the highest cumulative scores.The selected recognition result is determined after comparing the candidate outputs.
Experimental Results
The model achieves strong recognition across irregular and regular scene-text benchmarks, while its weakly supervised attention approximately localizes decoded characters. Failures remain under blur, occlusion, extreme distortion, uneven lighting, uncommon fonts, and vertical text.
- The approach outperforms all compared methods on irregular benchmarks and achieves comparable performance on regular text.
- 7.5% accuracy increase on SVTP-None, from 78.9% to 86.4%, demonstrates a substantial irregular-text gain.
- 10.1% accuracy increase on CT80, from 79.5% to 89.6%, further strengthens performance on irregular text.
- The model performs best or second best on 5 of 6 evaluated regular-text settings.
- Weakly supervised 2D attention approximately localizes decoded characters and extracts discriminative local features without character-level annotations.
- Failure cases include blurry images, partial occlusion, extreme distortion, uneven lighting, uncommon fonts, and vertical text.
Ablation Studies
Ablations show that sufficient CNN and LSTM capacity, real training data, and the proposed neighborhood-aware 2D attention contribute to recognition performance. The proposed attention also outperforms traditional 2D and 1D alternatives.
- Reducing convolutional channels by 50% lowers accuracy by 2 to 4 percentage points.
- A feature map with maximum size 6 × 40 supports encoding varied visual information, while further down-sampling produces worse performance.
- The proposed 2D attention outperforms traditional 2D attention by 1 to 2 percentage points through neighborhood information aggregation.
- Both 2D attention modules outperform the 1D module in most cases for regular and irregular text recognition.
- Reducing LSTM hidden-state size degrades accuracy, with one layer causing more serious degradation than two layers.
- Removing real image training data reduces performance, while the experiments are trained from scratch and evaluated without lexicons.
Conclusion
The paper presents a simple irregular-text recognition baseline built from standard neural modules and a tailored 2D attention mechanism. It reports robustness across regular and irregular layouts while identifying several extensions for future work.
- The framework combines a ResNet CNN, an LSTM encoder-decoder, and tailored 2D attention to select local character features without extra supervision.
- The approach performs well on both regular and irregular text layouts.
- Future extensions include CNN sequence modeling, graph-based attention with richer structures, and an auxiliary word-classification head.