Source-linked AI summary
A Multi-Object Rectified Attention Network for Scene Text Recognition
Canjie Luo, Lianwen Jin, Zenghui Sun
TL;DR
Irregular scene text is difficult to recognize because of varied shapes, distortions, and environmental disturbances. MORAN combines weakly supervised rectification with attention-based sequence recognition, fractional pickup, and curriculum learning, and reports outstanding performance across regular and irregular benchmarks. Its scope is limited to cropped text recognition and does not constitute an end-to-end system without a detector.
Problem
Irregular scene text remains difficult to recognize because varied shapes, distorted patterns, and environmental disturbances challenge current recognition models.
Method
MORAN combines a weakly supervised multi-object rectification network with an attention-based sequence recognition network, fractional pickup, and curriculum learning.
Results
MORAN demonstrates outstanding performance on regular and irregular benchmarks, including IIIT5K, SVT, ICDAR2003, ICDAR2013, ICDAR2015, SVT-Perspective, and CUTE80.
Takeaways & Limitations
Rectification makes irregular text more readable for attention-based recognition while the framework remains applicable to both regular and irregular text.
Takeaways & Limitations
Experiments use cropped text recognition; without a text detector, MORAN is not an end-to-end scene text recognition system, and it focuses more on horizontal irregular text.
Abstract
from arXiv · showhide
Irregular text is widely used. However, it is considerably difficult to recognize because of its various shapes and distorted patterns. In this paper, we thus propose a multi-object rectified attention network (MORAN) for general scene text recognition. The MORAN consists of a multi-object rectification network and an attention-based sequence recognition network. The multi-object rectification network is designed for rectifying images that contain irregular text. It decreases the difficulty of recognition and enables the attention-based sequence recognition network to more easily read irregular text. It is trained in a weak supervision way, thus requiring only images and corresponding text labels. The attention-based sequence recognition network focuses on target characters and sequentially outputs the predictions. Moreover, to improve the sensitivity of the attention-based sequence recognition network, a fractional pickup method is proposed for an attention-based decoder in the training phase. With the rectification mechanism, the MORAN can read both regular and irregular scene text. Extensive experiments on various benchmarks are conducted, which show that the MORAN achieves state-of-the-art performance. The source code is available.
1. Introduction
Scene text recognition remains challenging because environmental disturbances and irregular shapes distort text. MORAN addresses this by jointly using rectification, attention-based recognition, fractional pickup, and curriculum learning.
- Irregular shapes, distorted patterns, and environmental disturbances make scene text recognition especially challenging.
- MORAN separates recognition into MORN rectification and ASRN attention-based sequence recognition.MORN rectifies images before ASRN sequentially reads the text.
- MORN learns image offsets under weak supervision, requiring only text labels rather than geometric- or pixel-level supervision.It generates an offset grid and samples pixels to produce a rectified image.
- Curriculum learning first optimizes MORN and ASRN separately before end-to-end optimization, enabling more efficient training.The two subnetworks are mutually beneficial in performance.
- Fractional pickup randomly stretches different feature-map parts during training, improving attention sensitivity and robustness to context variation.The method expands the MORAN visual field to address noise perturbations.
- MORAN outperforms state-of-the-art methods on several standard benchmarks, including regular and irregular text datasets.The listed benchmarks include IIIT5K, SVT, ICDAR2003, ICDAR2013, ICDAR2015, SVT-Perspective, and CUTE80.
2. Related Work
Prior work for irregular text recognition follows bottom-up character localization or top-down text rectification. MORAN adopts the top-down route while adding fractional pickup and curriculum learning.
- Irregular text recognition is harder than regular recognition because text characters vary in shape, orientation, and scale.
- Bottom-up approaches: Bottom-up methods locate individual characters and connect them into text sequences.Examples use two-dimensional attention or positional information to extract character features.
- Top-down approaches: Top-down methods match text shape and rectify it to reduce recognition difficulty.STAR-Net uses affine transformation, while RARE uses fiducial points and thin-plate-spline transformation.
- MORAN: MORAN follows the top-down approach and uses fractional pickup to improve character-focused attention.Its curriculum learning strategy is intended to improve convergence during training.
3. Methodology
MORAN separates irregular scene text recognition into unconstrained image rectification and attention-based sequence recognition. Its weakly supervised MORN, fractional pickup training, and curriculum strategy make distorted text easier to recognize and improve training.
- Overall architecture: MORAN combines a multi-object rectification network (MORN) with an attention-based sequence recognition network (ASRN).The MORN rectifies distorted images before the ASRN reads the text.
- Multi-object rectification: Affine transformations are limited to rotation, scaling, and translation, so they cannot cover all complicated scene-text deformations.The paper reports slanted characters and black-edge noise after affine rectification.
- Multi-object rectification: MORN predicts position offsets for image parts without geometric constraints, then rectifies the image through sampling.Pooling is used before the convolutional layer to reduce noise and computation; the network predicts offsets rather than character categories.
- Multi-object rectification: MORN is trained with images and text labels only because its differentiable sampling operation supports back-propagation without deformation annotations.The offset grid is generated from normalized pixel coordinates, and bilinear interpolation produces the rectified image.
- Fractional pickup: Fractional pickup randomly stretches different feature-map regions during training so the attention decoder perceives neighboring features and becomes more robust to context variation.The method produces smoother α_t maps and expands the decoder’s visual field beyond target characters.
- Curriculum training: Curriculum learning first optimizes MORN and ASRN separately, then jointly trains them end to end because the sub-networks can hinder each other during direct training.The paper describes this three-step strategy as a way to improve training efficiency.
4. Experiments
Experiments evaluate MORAN on regular and irregular cropped-text benchmarks using word accuracy, implementation details, ablations, comparisons, and documented limitations. MORAN performs strongly across lexicon-free and irregular-text settings, while remaining constrained by its training data and cropped-horizontal input scope.
- Experimental setup: Experiments measure performance by word accuracy across regular and irregular text-recognition benchmarks, including IIIT5K, SVT, ICDAR, SVT-Perspective, and CUTE80.The evaluation uses cropped word images, with lexicon-free and lexicon-based settings where applicable.
- Ablation studies: A max-pooling layer with kernel size 2 and stride 1 gives the highest accuracy among the tested pooling configurations.The comparison includes no pooling, average pooling, and max pooling at the top of MORN.
- Ablation studies: Curriculum learning progressively combines ASRN, MORN-based rectification, and fractional pickup, with the complete MORAN trained end-to-end achieving promising performance.The staged experiments first train ASRN, then add MORN, and finally include fractional pickup.
- Comparisons: MORAN is more flexible than affine transformation because it predicts smooth rectification without geometric constraints on rotation, scaling, and translation.Affine transformation is described as unable to cover all complicated scene-text deformations.
- General benchmarks: MORAN outperforms current state-of-the-art methods in lexicon-free mode and outperforms all methods without a lexicon on SVT-Perspective.The comparison excludes settings considered outside scope, including word-category prediction and pixel-level supervision.
- Irregular text: MORAN rectifies most curved text in CUTE80 and correctly recognizes it, showing robustness to text with small curve angles.Figure 9 states that MORAN can rectify irregular text with small curve angles.
- Limitations: MORAN can fail on large curve angles because complicated backgrounds may be mistaken for foreground, although such samples are rare in training datasets.The training datasets contain only horizontal synthetic text, limiting coverage of these cases.
- Limitations: Without data augmentation, MORAN focuses more on horizontal irregular text and is not designed for vertical text.The stated method scope is complicated deformation within a cropped horizontal rectangle.
5. Conclusion
The paper concludes that MORAN combines unconstrained rectification with attention-based sequence recognition, fractional pickup, and curriculum learning for regular and irregular scene text. Experiments across multiple datasets demonstrate strong performance, while arbitrary-oriented recognition and integration with detectors remain future directions.
- Conclusion: MORAN uses rectification followed by attention-based sequence recognition to convert irregular text into a more readable form and output characters sequentially.The framework contains MORN for flexible rectification and ASRN for recognition.
- Conclusion: Fractional pickup expands the attention decoder’s visual field, providing more context information and improving robustness.The method changes feature-map regions at several stretch scales during training.
- Conclusion: MORAN is trained with weak supervision using only images and corresponding text labels, without geometric-level or pixel-level labels.Curriculum learning is used to strengthen the two subnetworks during training.
- Conclusion: Experiments on regular and irregular datasets, including IIIT5K, SVT, ICDAR2003, ICDAR2013, ICDAR2015, SVT-Perspective, and CUTE80, demonstrate outstanding performance.
- Future work: Future work includes arbitrary-oriented text recognition and combining MORAN with a scene-text detector for end-to-end recognition.The paper notes that arbitrary-oriented text is challenging because of varied text and background.