Source-linked AI summary
Robust Scene Text Recognition with Automatic Rectification
Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, Xiang Bai
TL;DR
Scene text recognition must handle natural-image text with irregular shapes, including perspective and curved layouts. RARE combines TPS-based spatial rectification with attention-based sequence recognition, and experiments report strong benchmark performance and improved recognition of irregular text, though heavy perspective distortion can still defeat the STN.
Problem
Natural-image text recognition remains challenging because scene text varies in appearance and may have perspective or curved shapes unlike regular document text.
Method
RARE combines a Spatial Transformer Network that predicts TPS rectification with an attention-based Sequence Recognition Network, trained end-to-end without geometric supervision.
Results
The experiments report state-of-the-art or highly competitive benchmark performance and significant accuracy improvements on irregular scene text.
Takeaways & Limitations
The learned rectifier can automatically generate more readable images for humans and the sequence recognizer across several irregular text types.
Takeaways & Limitations
The STN sometimes fails under heavy perspective distortion.
Abstract
from arXiv · showhide
Recognizing text in natural images is a challenging task with many unsolved problems. Different from those in documents, words in natural images often possess irregular shapes, which are caused by perspective distortion, curved character placement, etc. We propose RARE (Robust text recognizer with Automatic REctification), a recognition model that is robust to irregular text. RARE is a specially-designed deep neural network, which consists of a Spatial Transformer Network (STN) and a Sequence Recognition Network (SRN). In testing, an image is firstly rectified via a predicted Thin-Plate-Spline (TPS) transformation, into a more "readable" image for the following SRN, which recognizes text through a sequence recognition approach. We show that the model is able to recognize several types of irregular text, including perspective text and curved text. RARE is end-to-end trainable, requiring only images and associated text labels, making it convenient to train and deploy the model in practical systems. State-of-the-art or highly-competitive performance achieved on several benchmarks well demonstrates the effectiveness of the proposed model.
1. Introduction
RARE addresses scene text recognition when natural-image text is irregular rather than tightly bounded, horizontal, and frontal. It combines rectification with sequence recognition and trains both components end-to-end.
- Natural-image text remains challenging because illumination, motion blur, font, color, and irregular geometry vary substantially.
- RARE combines a Spatial Transformer Network with a Sequence Recognition Network to rectify input images before recognition.The STN uses a TPS transformation, while the SRN recognizes text as a character sequence.
- TPS nonlinearity enables rectification of perspective and curved text by transforming irregular inputs into images containing regular text.
- The SRN uses an encoder-decoder attention model whose decoder recurrently generates characters from sequential image features.
- The complete model can be trained end-to-end without geometric labels for TPS fiducial-point positions.The STN is supervised through error differentials back-propagated by the SRN.
- The paper contributes an irregular-text recognizer, an STN connected to attention-based recognition, and a convolutional-recurrent SRN encoder.
2. Related Work
Prior scene text recognition work includes bottom-up character-based methods, top-down whole-word or image methods, and sequence-recognition approaches. Irregular text has received comparatively less explicit attention, motivating rectification-based methods.
- Bottom-up methods detect individual characters using sliding windows, connected components, or Hough voting, then integrate them into words.
- Top-down methods recognize text from entire images, including label embeddings, large word-class classifiers, and structured-output CNNs.
- Sequence-recognition methods represent text as character sequences and predict those sequences from sequential image representations.
- Irregular text is relatively less addressed explicitly, although prior work studies multi-oriented text and rectifies character or perspective distortions.
3. Proposed Model
The proposed model maps an input image to a variable-length character sequence. Its output is the ordered string of predicted characters.
- RARE takes an input image I and outputs a character sequence l = (l1, . . . , lT), where T is the variable string length.
3.1. Spatial Transformer Network
The STN predicts a TPS transformation that maps input images to rectified images through fiducial points, a generated sampling grid, and differentiable sampling. This flexibility targets several forms of irregular text.
- Spatial Transformer Network: The STN predicts fiducial points, computes TPS parameters and a sampling grid, then samples the input image to produce a rectified image.
- Spatial Transformer Network: Differentiable localization, grid generation, and sampling allow recognition errors to back-propagate through the STN.
- 3.1.1 Localization Network: The localization network directly regresses the x,y-coordinates of K fiducial points from global image context.
- 3.1.1 Localization Network: Fiducial coordinates use a normalized image-centered coordinate system with xk and yk constrained to [−1, 1].
- 3.1.1 Localization Network: A CNN with a tanh output layer produces the 2K coordinate values, while no fiducial-point coordinates are annotated during training.
- 3.1.2 Grid Generator: The grid generator uses constant base fiducial points distributed along the top and bottom edges of the rectified image.
- 3.1.2 Grid Generator: For each rectified-image point, TPS finds a corresponding point on the input image, and the sampler bilinearly interpolates nearby pixel values.
- 3.1.3 Sampler: TPS can rectify loosely bounded, multi-oriented, perspective, and curved text into images that are more readable for the recognizer.
3.2. Sequence Recognition Network
The SRN recognizes rectified word images as character sequences using an encoder–decoder architecture. Convolutional features are organized into a sequence, contextualized with BLSTM, and decoded recurrently with attention.
- 3.2.1 Encoder: Convolutional-Recurrent Network: The encoder converts convolutional feature maps into a left-to-right sequence of vectors, each describing a local receptive field.The map-to-sequence operation extracts and flattens feature-map columns in left-to-right order.
- 3.2.1 Encoder: Convolutional-Recurrent Network: A two-layer BLSTM models dependencies in both directions and outputs the sequential representation h with the same sequence length.The representation is h = (h_1, . . . , h_L), where L equals the convolutional feature-map width.
- 3.2.2 Decoder: Recurrent Character Generator: The decoder recurrently generates characters conditioned on the encoder representation, using attention to select relevant contents at each step.Its recurrence uses a GRU cell and produces an EOS-terminated character sequence.
- 3.2. Sequence Recognition Network: The SRN directly maps a rectified image sequence to a character sequence, allowing input and output sequences to have arbitrary lengths.It can be trained using only word images and associated text labels.
3.3. Model Training
The model is trained by minimizing sequence negative log-likelihood, with both STN and SRN parameters optimized jointly. Specialized initialization of the localization network is important for convergence.
- 3.3. Model Training: Training minimizes the negative log-likelihood over paired training images and text labels.The probability distribution is computed by the decoder, while θ contains the parameters of both STN and SRN.
- 3.3. Model Training: The localization network’s output layer is initialized to produce a predefined fiducial-point pattern rather than being randomly initialized.The initial biases yield the pattern shown in the first initialization example.
- 3.3. Model Training: Random initialization of the localization network causes training to fail to converge.Alternative fiducial-point initialization patterns were empirically associated with poorer performance.
3.4. Recognizing With a Lexicon
Lexicon-constrained recognition selects words from a prefix tree rather than evaluating every lexicon word independently. Beam search can improve recognition performance at the cost of search speed.
- 3.4. Recognizing With a Lexicon: Very large lexicons make precise search time-consuming because it iterates over all lexicon words.The Hunspell lexicon is given as an example containing more than 50k words.
- 3.4. Recognizing With a Lexicon: A prefix tree shares computation among words with common prefixes and selects the highest-posterior child at each step until reaching a leaf.The resulting search requires much less computation than precise enumeration because its depth is bounded by the longest lexicon word.
- 3.4. Recognizing With a Lexicon: Beam search maintains multiple candidate nodes and retains the top-B paths by accumulated log-likelihood after each decoding step.Increasing beam width usually improves performance but reduces search speed.
4. Experiments
Experiments evaluate RARE on general and irregular-text benchmarks, alongside recognition speed and qualitative rectification behavior. The model performs strongly on unconstrained and irregular-text recognition, while large-lexicon search is slower and heavy perspective distortion can still challenge rectification.
- Implementation and speed: The model recognizes images in under 2ms without a lexicon, but a 50k-word lexicon requires approximately 200ms using approximate beam search.Lexicon size determines the search strategy: precise search is used up to 1k words, while larger lexicons use beam width 7.
- General benchmarks: RARE outperforms prior methods on unconstrained general-benchmark recognition, including nearly four percentage points over CRNN on IIIT5K.For constrained recognition, it achieves state-of-the-art or highly competitive accuracies, generally on par with or slightly below compared methods.
- Perspective text: On SVT-Perspective, RARE substantially outperforms the compared perspective-recognition method and CRNN across full-lexicon and lexicon-free settings.The comparison with CRNN uses the same training set, while the comparison with the perspective-specific method is partly affected by RARE’s larger training set.
- Perspective text: Qualitative results show that the STN places fiducial points along text edges and produces rectified images that are more readable for the SRN.The examples include input images, predicted fiducial points, rectified grayscale images, recognized text, and ground truth; some characters are marked as mistaken.
- Perspective text: The STN sometimes fails when perspective distortion is heavy, identifying a practical boundary of the rectification approach.This limitation is reported alongside the qualitative perspective-text analysis.
- Curved text: On CUTE80 curved-text recognition, RARE outperforms the two compared methods by a large margin.One baseline cannot recognize words outside its dictionary, while the other recognizes arbitrary words but lacks a mechanism specifically designed for curved text.
5. Conclusion
The conclusion presents RARE as an end-to-end approach to irregular scene text recognition that learns rectification without geometric supervision. Experiments support improved irregular-text accuracy and competitive overall recognition, while future work extends the system toward end-to-end scene text reading.
- 5. Conclusion: RARE addresses irregular scene text recognition by connecting a differentiable spatial transformer with an attention-based sequence recognizer.The combined model is trained end-to-end.
- 5. Conclusion: The learned model can generate more readable images without geometric supervision, using gradients propagated through the sequence recognizer.The conclusion reports this as an experimentally supported property of the system.
- 5. Conclusion: Experiments show that text rectification significantly improves recognition accuracy on irregular scene text and that the full system is competitive with state-of-the-art methods.The paper identifies combining RARE with a scene text detector as future work for end-to-end scene text reading.