Source-linked AI summary
ESIR: End-to-end Scene Text Recognition via Iterative Image Rectification
Fangneng Zhan, Shijian Lu
TL;DR
Scene text recognition remains challenged by perspective and curvature distortions despite progress on imaging artifacts and text styles. ESIR addresses these distortions with an end-to-end iterative rectification system using line-fitting transformation, and experiments report superior rectification and recognition performance across public datasets. The method also avoids extra annotations, while iterative rectification faces boundary-effect and image-clarity concerns.
Problem
Deep networks still face various problems recognizing scene texts with perspective distortions and text line curvature.
Method
ESIR uses an end-to-end rectification network with a line-fitting transformation and iterative distortion correction driven by recognition performance.
Results
ESIR achieves superior scene text rectification and recognition performance across a number of public datasets.
Takeaways & Limitations
The system corrects perspective and curvature distortions without extra annotations beyond scene text images and word-level annotations.
Takeaways & Limitations
Direct iterative rectification can accumulate boundary effects and degrade image clarity through repeated bilinear interpolations.
Abstract
from arXiv · showhide
Automated recognition of texts in scenes has been a research challenge for years, largely due to the arbitrary variation of text appearances in perspective distortion, text line curvature, text styles and different types of imaging artifacts. The recent deep networks are capable of learning robust representations with respect to imaging artifacts and text style changes, but still face various problems while dealing with scene texts with perspective and curvature distortions. This paper presents an end-to-end trainable scene text recognition system (ESIR) that iteratively removes perspective distortion and text line curvature as driven by better scene text recognition performance. An innovative rectification network is developed which employs a novel line-fitting transformation to estimate the pose of text lines in scenes. In addition, an iterative rectification pipeline is developed where scene text distortions are corrected iteratively towards a fronto-parallel view. The ESIR is also robust to parameter initialization and the training needs only scene text images and word-level annotations as required by most scene text recognition systems. Extensive experiments over a number of public datasets show that the proposed ESIR is capable of rectifying scene text distortions accurately, achieving superior recognition performance for both normal scene text images and those suffering from perspective and curvature distortions.
1. Introduction
ESIR targets scene text recognition failures caused by perspective and curvature distortions through end-to-end iterative rectification. Its line-fitting transformation and iterative pipeline are designed to improve distortion correction and recognition without extra annotations.
- Scene text recognition remains difficult because text appearance varies substantially and backgrounds and imaging artifacts complicate recognition.
- Deep learning methods still face problems recognizing scene texts with arbitrary perspective distortions and text line curvature.
- ESIR iteratively rectifies perspective and curvature distortions before passing the final rectified image to a recognition network.
- ESIR is end-to-end trainable, requires no extra annotations beyond scene text images and word-level annotations, and is reported to improve recognition across public datasets.
- The line-fitting transformation models text-line pose with a polynomial for the middle line and line segments for vertical orientation and boundaries.
- Each rectification iteration further estimates and corrects distortions from the previously rectified image, guided by higher recognition accuracy.
2. Related Work
Scene text recognition methods include bottom-up character-based systems and top-down word- or line-level systems. Although RNN- and attention-based methods handle horizontal or slightly distorted text well, distortions involving perspective and curvature remain challenging, motivating iterative rectification.
- Bottom-up systems detect and recognize individual characters before linking them into words or text lines.
- Top-down systems recognize complete words or text lines directly without explicitly detecting and recognizing individual characters.
- RNNs encode words or text lines as feature sequences and perform recognition without character segmentation.
- RNN- and attention-based methods achieve strong results on horizontal or slightly distorted text but still struggle with perspective distortions and text-line curvatures.
- Earlier rectification approaches had limited success because they used hand-crafted features and required character-level information.
- ESIR differs from existing rectification methods by correcting distortions iteratively and introducing a flexible line-fitting transformation.
3. The Proposed Method
The proposed method combines a line-fitting transformation with iterative rectification to correct perspective and curvature distortions before sequence recognition. Its training is recognition-driven, uses differentiable sampling, and addresses initialization and iterative boundary effects.
- 3.1.1 Line-Fitting Transformation: ESIR estimates scene-text pose with a line-fitting transformation that models the middle line polynomially and uses line segments for vertical orientation and boundaries.The parameter count is 3L + K + 1 when L line segments and a middle-line polynomial of order K are used.
- 3.1.1 Line-Fitting Transformation: A localization network predicts fitting-line parameters, which determine thin plate spline control points for differentiable rectification.The sampler bilinearly interpolates pixels and can back-propagate image gradients.
- 3.1.2 Iterative Rectification: The rectification network repeatedly processes the rectified image, refining distortion estimates until a predefined iteration count before recognition.The final rectified image is fed to the sequence recognition network.
- 3.1.2 Iterative Rectification: The architecture uses intermediate rectified images for parameter estimation while feeding the original image to transformation stages, limiting accumulated boundary loss and interpolation blur.Direct iterative rectification can discard pixels outside the sampling region and degrade clarity through repeated bilinear interpolation.
- 3.2. Recognition Network: The recognition network combines a residual convolutional encoder, bidirectional LSTMs, and an attention-based sequence decoder.The input is resized to 32×100 pixels before feature extraction.
- 3.3. Network Training: To stabilize training, ESIR initializes P through P = P0 + ΔP and iteratively predicts small parameter updates rather than directly predicting P.This avoids highly distorted initial images and makes rectification training smoother and more stable.
4.1. Datasets and Metrics
Experiments use synthetic training data and six public scene-text datasets spanning mostly horizontal text and perspective- or curvature-distorted text. Evaluation follows established recognition protocols with lexicon-dependent decoding and correctly recognized words.
- Datasets: ESIR models are trained on Synth90K and SynthText without fine-tuning on a third dataset, then evaluated across six public datasets.The evaluation includes ICDAR2013, IIIT5K, SVT, ICDAR2015, SVTP, and CUTE80.
- Datasets: The benchmark separates three mostly horizontal datasets from three datasets containing substantial perspective or curvature distortions.The normal group is ICDAR2013, IIIT5K, and SVT; the distorted group is ICDAR2015, SVTP, and CUTE80.
- Training Data: Synth90K provides 9 million synthetic text images, while the study crops 4 million text patches from SynthText for training.Synth90K has a 90K lexicon, and the SynthText crop count is described as the lower end for fair benchmarking.
- Metrics: Evaluation covers 68 characters, counts only digits and letters, applies lexicon words with minimum edit distance when available, and reports correctly recognized words.Correctly recognized words are determined from the ground-truth transcription.
- Metrics: Table 2 reports recognition performance for ICDAR2013, ICDAR2015, IIIT5K, SVT, SVTP, and CUTE under different lexicon sizes or no lexicon.Method labels include the network backbone and training datasets, with SK denoting Synth90K and ST denoting SynthText.
4.2. Implementation
The implementation trains three ESIR variants under shared rectification settings, varying backbone and training-data combinations for benchmarking.
- Implementation: The recognition network is implemented in TensorFlow and trained with ADADELTA, weighted cross-entropy, one million iterations, and batch size 64.Training used a workstation with an Intel Core i7-7700K CPU and NVIDIA GeForce GTX 1080 Ti GPU.
- Model Variants: Three ESIR variants compare VGG and ResNet backbones and Synth90K-only versus combined Synth90K and SynthText training data.The combined-data ResNet model is intended for comparison with methods using both datasets.
- Model Variants: All three ESIR models use five rectification iterations, 20 line segments, and a fourth-order middle-line polynomial.These settings are shared across the evaluated variants.
4.3. Experimental Results
Across six public datasets, ESIR achieves strong scene-text recognition, including distorted images, while iterative rectification improves results with modest computational overhead. Visual comparisons and ablations support the value of iterative correction and line-segment modeling.
- Overall recognition performance: ESIR’s ResNet backbone consistently outperforms its VGG backbone across all six evaluated datasets.The authors attribute this result to the more powerful network backbone.
- Overall recognition performance: ESIR achieves superior scene text recognition performance across six public datasets compared with state-of-the-art techniques.It outperforms state-of-the-art methods on all three distorted datasets and also achieves state-of-the-art performance on the three normal datasets.
- Rectification and recognition: Visual comparisons show ESIR producing better rectifications than RARE and ASTER on distorted SVTP and CUTE80 examples.Figure 5 includes perspective and curvature distortions with complex image backgrounds.
- Ablation analysis: ESIR’s iterative rectification improves recognition consistently as the number of iterations increases, especially during the first two iterations.The effect is clearer on the highly distorted SVTP and CUTE datasets.
- Ablation analysis: Increasing the number of line segments also improves recognition, although less substantially than increasing rectification iterations.The authors conjecture that more line segments improve text-line pose estimation and subsequent rectification.
- Computational costs: With five rectification iterations, ESIR takes 3ms per image in training and 28ms per image in testing, versus ASTER’s 2.4ms and 20ms.The similar overall cost is attributed to ESIR’s small, computationally light rectification network.
5. Conclusions
ESIR is an end-to-end trainable scene-text recognizer that iteratively corrects perspective and curvature distortions without extra annotations. Experiments demonstrate superior rectification and recognition performance, while joint optimization with detection remains future work.
- Conclusion: ESIR iteratively estimates and corrects perspective distortion and text-line curvature, using recognition performance to drive rectification.Its line-fitting transformation estimates text-line pose, and the system requires no extra annotations beyond scene-text images and word-level annotations.
- Conclusion: Experiments on public datasets demonstrate superior scene-text rectification and recognition performance.
- Future work: Joint optimization with a detection model for an end-to-end scene-text reading system remains future work.