Source-linked AI summary
AON: Towards Arbitrarily-Oriented Text Recognition
Zhanzhan Cheng, Yangliu Xu, Fan Bai, Yi Niu, Shiliang Pu, Shuigeng Zhou
TL;DR
Irregular scene text remains difficult for methods designed around regular horizontal and frontal layouts. This paper introduces AON, which combines four-direction visual features and learned placement clues through a filter gate before attention-based decoding. It reports state-of-the-art performance on irregular benchmarks and comparable performance on regular ones.
Problem
Natural-image text recognition is difficult because scene text is often irregular, while existing methods mainly handle regular horizontal and frontal text.
Method
AON extracts four-direction features and character placement clues, fuses them with a filter gate, and feeds the integrated sequence to an attention-based decoder.
Results
The method achieves state-of-the-art performance on irregular benchmarks and is comparable to major existing methods on regular benchmarks.
Takeaways & Limitations
AON directly represents irregular text orientations and can be trained end-to-end using images with word-level annotations.
Takeaways & Limitations
Introducing AON doubles the parameters and computational cost of the naive base model, although STN requires triple the naive cost.
Abstract
from arXiv · showhide
Recognizing text from natural images is a hot research topic in computer vision due to its various applications. Despite the enduring research of several decades on optical character recognition (OCR), recognizing texts from natural images is still a challenging task. This is because scene texts are often in irregular (e.g. curved, arbitrarily-oriented or seriously distorted) arrangements, which have not yet been well addressed in the literature. Existing methods on text recognition mainly work with regular (horizontal and frontal) texts and cannot be trivially generalized to handle irregular texts. In this paper, we develop the arbitrary orientation network (AON) to directly capture the deep features of irregular texts, which are combined into an attention-based decoder to generate character sequence. The whole network can be trained end-to-end by using only images and word-level annotations. Extensive experiments on various benchmarks, including the CUTE80, SVT-Perspective, IIIT5k, SVT and ICDAR datasets, show that the proposed AON-based method achieves the-state-of-the-art performance in irregular datasets, and is comparable to major existing methods in regular datasets.
1. Introduction
Scene text recognition remains difficult because natural-image text is often irregular, while existing methods mainly assume regular horizontal, frontal arrangements. AON addresses this by combining four-direction features and placement clues with a filter gate and attention decoder, achieving strong performance across irregular and regular benchmarks.
- Natural-image text recognition is challenging because text can be arbitrarily oriented, curved, slanted, perspective-distorted, blurred, or unevenly illuminated.
- Unlike related approaches that use rectification or character-level supervision, the integrated AON framework trains end-to-end without character-level bounding-box annotations.
- Most existing recognizers treat text as a one-dimensional sequence in a default direction, limiting their ability to handle irregular character arrangements.
- AON extracts horizontal, reversed-horizontal, vertical, and reversed-vertical features, while learned character placement clues weight these sequences.
- The filter gate fuses the four directional features with placement clues into an integrated sequence for attention-based character decoding.
- AON achieves state-of-the-art performance on irregular benchmarks and performance comparable to major existing methods on regular benchmarks.
2. Related works
Scene text recognition methods include bottom-up character pipelines and top-down whole-text prediction, with sequence recognition becoming a common top-down formulation. For irregular text, prior rectification and auxiliary detection approaches have important optimization or annotation burdens, motivating AON's direct four-direction representation and end-to-end training.
- Top-down methods predict entire text from an image, while bottom-up methods detect and recognize characters before assembling words.
- Sequence-recognition methods encode images and text as feature and character sequences, but one-dimensional image encodings struggle with irregular text.
- STN-based methods rectify irregular text before recognition, yet training is difficult for curved, arbitrarily oriented, or perspective-distorted scenes without geometric supervision.
- AON instead extracts four-direction deep features, uses a filter gate to form an integrated sequence, and trains the recognition network end-to-end with word-level annotations.
3. The Framework
The framework combines foundational visual features, four-direction processing, filter-gated integration, and attention-based decoding into one end-to-end recognition network.
- Framework Overview: The complete framework combines BCNN, AON, FG, and an attention-based decoder to predict character sequences from text images.BCNN extracts low-level visual features; AON generates directional features and placement clues; FG integrates them before decoding.
- Framework Overview: Higher-level BCNN representations are used as AON’s initial state because they empirically improve performance.BCNN also reduces computational cost and graphics memory usage.
- Attention-based Decoder: The attention decoder generates variable-length character sequences from an integrated feature sequence using recurrent hidden states and attention weights.An EOS token terminates generation, allowing the decoder to produce sequences of different lengths.
- Network Training: The network is trained jointly from input images and ground-truth character sequences with a single loss over predicted characters.The parameters θ include all network components, and each target character contributes to the training objective.
- Character Sequence Decoding: Decoding supports both lexicon-free recognition and lexicon-constrained recognition using 50-word, 1k-word, or full lexicons.Lexicon-free decoding selects the most probable character directly, whereas constrained decoding selects the highest-probability lexicon word.
4. Technical Details of AON and FG
AON represents text in four reading directions and estimates character-placement clues, while FG uses those clues to integrate the relevant directional features.
- Arbitrary Orientation Network (AON): AON extracts horizontal, reversed-horizontal, vertical, reversed-vertical, and character-placement features through dedicated networks.HN and VN produce directional sequences, while CN outputs placement clues for the four directions.
- Character Placement Clues: Character-placement clues assign direction weights that guide the fusion of the four directional feature sequences.For each sequence position, the directional weights are normalized so their sum equals one.
- Shared Convolution Mechanism: A shared convolution mechanism applies the same filters to horizontal and vertical processing, improving robustness on orientation-unbalanced training data.The shared block is designed to make the two branches easier to learn when orientation distributions are unbalanced.
- Filter Gate (FG): The filter gate attends to the appropriate directional features using the corresponding placement clue and produces an integrated feature sequence.It suppresses irrelevant directional features before the sequence is passed to the decoder.
5. Performance Evaluation
AON is evaluated on irregular and regular recognition benchmarks using matched baselines and extensive experiments. Its directional features, placement clues, and fusion mechanism support strong irregular-text performance while remaining comparable on regular text.
- 5.3. Performance on Irregular Datasets: AON outperforms existing methods on almost all irregular benchmarks, except lexicon-free SVT-Perspective, while requiring only word-level annotations.Yang et al.'s method uses both word- and character-level bounding boxes, whereas AON avoids character-level annotations.
- 5.3. Performance on Irregular Datasets: TPS-based STN fails to satisfactorily rectify arbitrarily-oriented or seriously curved texts, with only two of six shown examples desirably rectified.The cited examples are shown in Figure 5.
- 5.4. Performance on Regular Datasets: AON achieves comparable performance to existing methods on regular benchmarks and surpasses Cheng's baseline in most cases.Cheng et al.'s strongest results use character-location annotations and a ResNet-based extractor, while AON is trained without character annotations.
- 5.5. Deep insight into AON: AON combines horizontal and vertical feature sequences for perspective, slant, and arbitrarily-oriented text, while HN and VN generate the directional features.CN learns character-placement weights, and FG fuses the directional features into the integrated feature sequence.
- 5.5. Deep insight into AON: Character-position distributions combine attention alignments with four placement clues, then normalize directional weights to estimate character coordinates and visualize text trends.The resulting connected arrows generally conform to visual observations in real images.
6. Discussions
The discussions show that CN is important for AON, aspect-ratio changes do not clearly harm recognition, and AON trades higher cost for less overhead than STN.
- The necessity of CN in AON: About 4% lower results across all benchmarks occur when horizontal and vertical features are concatenated temporally without CN.Channel-axis concatenation also converges slowly because three quarters of the final feature sequence is superfluous.
- Impact of aspect ratio: Large aspect ratios produce no obvious recognition degradation, including for horizontal texts after height enlarging or shrinking.This behavior is reported relative to previous works [32].
- Integrating with only two directional feature sequence: Using only right-left and down-top directional sequences can frame-wise mix the visual features of ‘p’ and ‘d’ through the filter gate.The example shows why integrating only two directional sequences is not reasonable.
- The computational cost of AON: AON doubles the Naive base model’s parameters and computational cost, while the STN base requires triple those resources.The comparison is motivated by resource-constrained settings such as embedded computer systems.
7. Conclusion
The paper proposes AON for arbitrarily oriented text recognition by combining four-direction visual features and placement clues with an attention-based decoder. Experiments validate its superiority on both irregular and regular benchmarks, while future work targets related tasks.
- 7. Conclusion: AON extracts four-direction character features and placement clues, fuses them with a filter gate, and decodes character sequences with attention.The method is presented as a unified approach to arbitrarily oriented text recognition.
- 7. Conclusion: Experiments over regular and irregular benchmarks validate the proposed method’s superiority within the evaluated settings.The paper also identifies extending the idea to related tasks as future work.