Source-linked AI summary
RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition
Xiaoyu Yue, Zhanghui Kuang, Chenhao Lin, Hongbin Sun, Wayne Zhang
TL;DR
Attention-based encoder-decoder recognition performs poorly on contextless text because decoding relies on contextual information that can drift. RobustScanner adds positional enhancement and dynamically fuses positional and contextual decoder features, achieving state-of-the-art regular and irregular benchmark results with little performance drop on contextless benchmarks.
Problem
Attention-based encoder-decoder methods perform poorly on contextless text sequences, limiting their robustness despite strong scene-text recognition results.
Method
RobustScanner adds a position enhancement branch and an element-wise dynamic fusion module to adaptively combine positional and contextual decoder clues.
Results
RobustScanner achieves new state-of-the-art results on popular regular and irregular benchmarks without much performance drop on contextless RandText.
Takeaways & Limitations
The method is robust across contextual and contextless text-recognition scenarios by dynamically emphasizing positional clues when contextual information is unreliable.
Abstract
from arXiv · showhide
The attention-based encoder-decoder framework has recently achieved impressive results for scene text recognition, and many variants have emerged with improvements in recognition quality. However, it performs poorly on contextless texts (e.g., random character sequences) which is unacceptable in most of real application scenarios. In this paper, we first deeply investigate the decoding process of the decoder. We empirically find that a representative character-level sequence decoder utilizes not only context information but also positional information. Contextual information, which the existing approaches heavily rely on, causes the problem of attention drift. To suppress such side-effect, we propose a novel position enhancement branch, and dynamically fuse its outputs with those of the decoder attention module for scene text recognition. Specifically, it contains a position aware module to enable the encoder to output feature vectors encoding their own spatial positions, and an attention module to estimate glimpses using the positional clue (i.e., the current decoding time step) only. The dynamic fusion is conducted for more robust feature via an element-wise gate mechanism. Theoretically, our proposed method, dubbed \emph{RobustScanner}, decodes individual characters with dynamic ratio between context and positional clues, and utilizes more positional ones when the decoding sequences with scarce context, and thus is robust and practical. Empirically, it has achieved new state-of-the-art results on popular regular and irregular text recognition benchmarks while without much performance drop on contextless benchmarks, validating its robustness in both contextual and contextless application scenarios.
1 Introduction
RobustScanner addresses contextless scene-text recognition by analyzing how decoder queries combine positional and contextual information. It enhances positional clues and dynamically fuses them with conventional decoder features, achieving strong benchmark performance while remaining robust on contextless text.
- Scene text recognition remains challenging because text can be curved, oriented, or distorted, creating misalignment between output sequences and two-dimensional input images.
- The attention-based encoder-decoder performs poorly on contextless text sequences despite its effectiveness on academic benchmarks.Each decoding step uses an LSTM query, attention-derived glimpse, and classifier to predict a character or <EOS>.
- Decoder query vectors encode both context and positional information, but contextual information dominates at later decoding steps and can cause misrecognition on contextless images.
- RobustScanner adds a position enhancement branch that estimates glimpses from the current decoding time step and a dynamic fusion module that combines positional and conventional decoder outputs.An element-wise gate adaptively adjusts the ratio between positional and context information at each time step.
- RobustScanner achieves new state-of-the-art performance on regular and irregular benchmarks without much performance drop on contextless RandText.
2 Related Work
Scene-text recognition methods have progressed from character detection and rectification toward end-to-end encoder-decoder approaches. RobustScanner extends attention-based recognition by dynamically balancing positional and contextual clues to address alignment drift.
- Traditional bottom-up methods detect individual characters before combining them, but can miss small characters and suffer from background clutter, illumination, blur, and noise.
- Rectification-based approaches: Rectification-based approaches transform irregular text images into regular ones before recognition, using increasingly flexible or iterative transformations.
- Segmentation-based approaches: Segmentation-based approaches recognize individually segmented characters, but commonly require character-level annotations unavailable in most public datasets.
- Encoder-decoder with attention-based approaches: Encoder-decoder attention methods became state-of-the-art approaches for irregular text recognition by predicting text sequences end to end.
- Encoder-decoder with attention-based approaches: Unlike approaches that decouple attention from historical decoding results, RobustScanner dynamically adjusts the ratio of positional and contextual clues during decoding.
3 Methodology
RobustScanner analyzes how attention-based decoding combines context and positional information, then adds a positional branch and dynamically fuses it with the conventional decoder. Its modules produce position-aware features and adaptively combine contextual and positional glimpses during character prediction.
- Background: The conventional decoder uses an LSTM sequence module, attention module, and prediction module to generate one character or <EOS> token per decoding step.The LSTM receives the previously predicted character and hidden state, attention computes a glimpse, and the prediction module classifies it.
- Decoder Dissection: Query vectors encode positional information as well as context, with same-position queries remaining similar across different text sequences.The first query vector remains unchanged across sequences despite differing characters, indicating that it does not encode context information.
- Decoder Dissection: The decoder’s contextual information becomes stronger and positional information weaker at later time steps, which can produce alignment drift on contextless text.The similarity analysis and regression study examine how query vectors correspond to character positions across sequences.
- RobustScanner: RobustScanner adds a position enhancement branch that uses time-step embeddings, position-aware encoder features, and an attention module to estimate positional glimpses.The position embedding is constant for a given decoding step across sequences, while the position-aware module uses global information to produce position-aware feature maps.
- RobustScanner: The position-aware module processes the encoder feature map with two LSTM layers and convolutional operations before supplying position-aware features to attention.The module outputs F2 through recurrent processing and applies a stack of 3 × 3 convolution, ReLU, and 3 × 3 convolution operations.
- RobustScanner: A dynamically-fusing module combines hybrid and positional glimpses using an element-wise gate that predicts per-dimension attention weights.The fused glimpse is formed from the outputs of the contextual hybrid branch and the position enhancement branch before prediction.
4 Experiments
RobustScanner is evaluated on regular, irregular, contextless, and license-plate text recognition settings, with ablations isolating its positional and fusion components. It generally improves over prior methods and remains effective when contextual clues are scarce.
- Datasets and Evaluation: The experiments use six standard benchmarks spanning regular and irregular text, plus RandText and a license-plate test set.Training follows prior settings with MJSynth and SynthText, while the license-plate experiment tests domain differences in background, font, and layout.
- Comparison with State-of-the-art Approaches: RobustScanner achieves the best results on four datasets using only synthetic training data and outperforms SAR on four of six benchmarks.It reaches 92.4% accuracy on CUTE 80, while performing worse than SAR on SVT and SVTP, where contextual words and degraded images make context more important.
- Comparison with State-of-the-art Approaches: RobustScanner substantially outperforms representative attention-based methods on RandText, with a 4.8% absolute advantage over DAN.RandText contains 500 images with random letters and numbers pasted onto white backgrounds.
- Ablation Study: Removing the position enhancement branch reduces RandText accuracy to 46.8%, 34.4% below the complete model.The ablation shows that positional encoding is important on both contextual and contextless benchmarks.
- Ablation Study: The position aware module improves accuracy across IIIT5K, SVT, ICDAR 2013, ICDAR 2015, SVTP, CUTE 80, and RandText.Without it, the reported accuracies decrease from 92.5%, 84.4%, 91.4%, 70.7%, 74.4%, 83.3%, and 83.4% to 88.7%, 84.1%, 89.0%, 61.8%, 62.3%, 76.7%, and 68.0%, respectively.
- Ablation Study: Dynamic fusion achieves the best results on six academic benchmarks and the second-best result on RandText among the compared fusion methods.Unlike addition and concatenation, dynamic fusion changes across decoding steps and sequences.
5 Conclusions
The paper identifies weakened positional clues and increasingly dominant contextual clues as a source of contextless-text misrecognition, then introduces RobustScanner to dynamically combine both. Experiments and ablations support its effectiveness across regular, irregular, and contextless text benchmarks.
- Conclusion: RobustScanner targets misrecognition on contextless text by dynamically fusing hybrid and position enhancement branches during decoding.Its position aware module strengthens the positional encoding capacity of the position enhancement branch.
- Conclusion: The decoder’s positional clues weaken while contextual clues strengthen as decoding advances, motivating adaptive clue weighting.The paper reports ablations validating the effects of the proposed components.
- Conclusion: RobustScanner achieves state-of-the-art results on popular regular and irregular benchmarks without much performance drop on contextless benchmarks.The conclusion presents this combination as evidence of robustness across contextual and contextless settings.