Source-linked AI summary
Scene Text Recognition from Two-Dimensional Perspective
Minghui Liao, Jian Zhang, Zhaoyi Wan, Fengming Xie, Jiajun Liang, Pengyuan Lyu, Cong Yao, Xiang Bai
TL;DR
The paper addresses the mismatch between one-dimensional sequence recognition and the two-dimensional distribution of scene text. It proposes CA-FCN, which performs pixel-level character prediction and word formation in two-dimensional space. Experiments report strong performance on regular and irregular datasets and greater robustness to imprecise text localization.
Problem
Sequence-based scene text recognition overlooks text’s two-dimensional spatial distribution, potentially losing useful information and introducing noise.
Method
CA-FCN predicts characters at pixel level with character attention, then a word formation module groups and arranges pixels into words.
Results
The method outperforms previous methods on regular and irregular text datasets and is more robust to imprecise text localization.
Takeaways & Limitations
Two-dimensional character prediction can recognize regular and irregular text while simultaneously providing each character’s position.
Abstract
from arXiv · showhide
Inspired by speech recognition, recent state-of-the-art algorithms mostly consider scene text recognition as a sequence prediction problem. Though achieving excellent performance, these methods usually neglect an important fact that text in images are actually distributed in two-dimensional space. It is a nature quite different from that of speech, which is essentially a one-dimensional signal. In principle, directly compressing features of text into a one-dimensional form may lose useful information and introduce extra noise. In this paper, we approach scene text recognition from a two-dimensional perspective. A simple yet effective model, called Character Attention Fully Convolutional Network (CA-FCN), is devised for recognizing the text of arbitrary shapes. Scene text recognition is realized with a semantic segmentation network, where an attention mechanism for characters is adopted. Combined with a word formation module, CA-FCN can simultaneously recognize the script and predict the position of each character. Experiments demonstrate that the proposed algorithm outperforms previous methods on both regular and irregular text datasets. Moreover, it is proven to be more robust to imprecise localizations in the text detection phase, which are very common in practice.
Introduction
Scene text recognition remains challenging because text varies widely in shape and appearance, while sequence-based methods compress inherently two-dimensional text into one-dimensional features. CA-FCN instead predicts characters in two-dimensional space and forms words with their character locations.
- Scene text recognition supports applications including driverless-vehicle sign reading, human-computer interaction, assistive technologies, and guide-board recognition.
- Recent methods encode images into feature sequences and decode them with RNN or CTC, performing well mainly on horizontal or nearly horizontal text.
- Characters in scene images may be scattered, arbitrarily oriented, or curved, so one-dimensional encoding can lose key information or introduce undesired noise.
- CA-FCN predicts characters at pixel level in two-dimensional space, then a word formation module produces the word and the location of each character.
- The method is reported to be more robust to varied text shapes, background noise, and imprecise detection-stage localization, using character annotations obtained from public synthetic data.
- The paper reports state-of-the-art performance on regular datasets, a large margin over existing methods on irregular datasets, and robustness to imprecise localization.
Related Work
Earlier scene text systems used character detection or encoder-decoder sequence recognition, but one-dimensional representations are poorly matched to curved or seriously distorted text. The paper positions CA-FCN as a two-dimensional alternative that avoids these disadvantages.
- Traditional systems detect characters through binarization or sliding-window operations before recognizing them as words.
- Binarization-based methods struggle with varying backgrounds, fonts, colors, and uneven illumination in natural scene images.
- Encoder-decoder methods encode images into feature sequences and decode characters with RNN or CNN networks followed by CTC word formation.
- Because scene text is two-dimensional, compressing it into one-dimensional sequences may lose key information and add noise, especially for curved or seriously distorted text.
- Prior attention-based encoder-decoder systems can suffer training confusion from misalignment between ground-truth strings and attention output sequences caused by missing or superfluous characters.
- CA-FCN uses a two-dimensional character-attention FCN, achieving high accuracy on regular and irregular text while remaining robust to imprecise detection localization.
Methodology
CA-FCN recognizes scene text in two dimensions through pixel-level character prediction, attention-guided feature refinement, deformable convolutions, and word formation.
- Architecture: The architecture combines a Character Attention FCN for pixel-level character prediction with a word formation module that groups pixels into words.The word formation module converts two-dimensional character maps into character sequences.
- Architecture: CA-FCN predicts characters in a two-dimensional output map whose classes include character categories and background.Its output has spatial dimensions tied to the input image and supports varied text shapes.
- Character attention: Character attention highlights foreground characters, weakens background regions, and separates adjacent characters using a two-class attention map.The attention map is broadcast to the input feature-map shape for element-wise multiplication.
- Deformable convolution: Deformable convolution learns kernel offsets to provide flexible receptive fields for character prediction, improving handling of extra background from inaccurate detections.It is applied in stages 4 and 5, followed by a 3 × 1 convolution.
- Training: Character-region labels are generated from original character bounding boxes, with shrink ratios of 0.5 for attention supervision and 0.25 for final-output supervision.Shrinking reduces overlap between adjacent characters and simplifies word formation.
- Training: Training minimizes a weighted sum of character prediction loss and character attention loss, with α empirically set to 1.0.The prediction loss uses pixel weights, while the attention loss is binary cross entropy over character and background labels.
Experiments
Experiments evaluate CA-FCN on synthetic training data and regular, irregular, and expanded text benchmarks. The method achieves strong recognition accuracy, handles varied text shapes, and is more robust to imprecise text localization than sequence-based models.
- Experimental setup: The model is trained purely on synthetic data and evaluated without fine-tuning on four regular and irregular text benchmarks.SynthText supplies about 7 million recognition images with character-level annotations.
- Experimental setup: CA-FCN handles varied text shapes, including the curved text represented in CUTE.CUTE contains 288 images with substantial curved text and highly varying shapes.
- Benchmark results: 3.7 percent improvement over the previous state of the art is achieved on IIIT without lexicons, while CUTE gains 3.1 percent without extra curved-text training data.Comparable results are reported on SVT and IC13.
- Benchmark results: The method outperforms the compared approach on all benchmarks when evaluated against Cheng et al. (2018), particularly for irregular shapes such as curves.The comparison concerns a method using four adaptively directed one-dimensional feature sequences.
- Robustness to localization errors: On expanded images, CA-FCN is more robust than sequence-based models; its IIIT-p gap ratio is 2.6%, compared with 6.4% for CRNN.Sequence-based models often predict extra characters after image expansion, whereas CA-FCN remains stable.
- Robustness to localization errors: The deformable and attention modules both improve performance, while deformable convolution also contributes to robustness on expanded datasets.The proposed explanation is that two-dimensional prediction is less likely to encode extra background as misleading sequence features.
Conclusion
CA-FCN models scene text recognition in two dimensions, enabling recognition of both regular and irregular text while remaining robust to imprecise localization.
- CA-FCN models scene text recognition in a two-dimensional fashion through character classification at each pixel location.
- The algorithm effectively recognizes irregular as well as regular text instances.
- CA-FCN outperforms existing methods on datasets containing regular and irregular text.
- The method is much more robust to imprecise text localization than existing text recognition algorithms.