Source-linked AI summary
SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition
Zhi Qiao, Yu Zhou, Dongbao Yang, Yucan Zhou, Weiping Wang
TL;DR
Scene text recognition methods remain vulnerable to low-quality images because local visual sequence modeling does not explicitly capture global word semantics. SEED predicts semantic information from visual features, supervises it with pretrained word embeddings, and uses it to initialize and guide decoding. Integrated with ASTER, the framework reports improved robustness and state-of-the-art results on several benchmark datasets, especially low-quality ones.
Problem
Existing scene text recognition methods can struggle with blur, occlusion, background interference, and incomplete characters because they define recognition locally and overlook global word information.
Method
SEED predicts additional global semantic information from visual features, supervises it with pretrained word embeddings, and feeds it into the decoder to guide recognition.
Results
SEED achieves state-of-the-art performance on several standard benchmarks, especially the low-quality ICDAR2015 and SVT-Perspective datasets, with greater robustness to incomplete characters.
Takeaways & Limitations
Using predicted semantics as decoder initialization can improve recognition accuracy, particularly for low-quality scene text images.
Takeaways & Limitations
The framework's reported performance depends on the chosen baseline, and the authors expect stronger results when integrating a stronger baseline model.
Abstract
from arXiv · showhide
Scene text recognition is a hot research topic in computer vision. Recently, many recognition methods based on the encoder-decoder framework have been proposed, and they can handle scene texts of perspective distortion and curve shape. Nevertheless, they still face lots of challenges like image blur, uneven illumination, and incomplete characters. We argue that most encoder-decoder methods are based on local visual features without explicit global semantic information. In this work, we propose a semantics enhanced encoder-decoder framework to robustly recognize low-quality scene texts. The semantic information is used both in the encoder module for supervision and in the decoder module for initializing. In particular, the state-of-the art ASTER method is integrated into the proposed framework as an exemplar. Extensive experiments demonstrate that the proposed framework is more robust for low-quality text images, and achieves state-of-the-art results on several benchmark datasets.
1. Introduction
Scene text recognition remains difficult in low-quality images because existing methods often model characters locally while overlooking global word semantics. SEED adds predicted semantic information, supervised by pretrained word embeddings and used to guide decoding, achieving especially robust recognition on challenging benchmarks.
- Scene text recognition remains challenging despite progress, particularly under blur, background interference, occlusion, and incomplete characters.
- Existing encoder-decoder methods formulate recognition as local sequence character classification and can overlook global information across the whole word.
- SEED predicts additional global semantic information to guide decoding and supervises it with word embeddings from a pretrained language model.The semantic representation is intended to bridge visual features and language information during recognition.
- SEED integrates the state-of-the-art ASTER method as an exemplar framework.
- Extensive experiments report state-of-the-art performance on several public benchmarks, especially the low-quality ICDAR2015 and SVT-Perspective datasets.The reported framework is particularly more robust for incomplete characters.
2. Related Work
Scene text recognition research has progressed from handcrafted and classification-based systems toward deep encoder-decoder, CTC, attention, and rectification methods. Related semantic approaches exist, but the paper identifies limited explicit and effective integration of semantics into recognition.
- Traditional recognition methods detect and classify characters before grouping them into words or text lines using handcrafted features, classifiers, rules, language models, or lexicons.
- Deep learning methods increasingly recognize words or text lines directly, with sequence prediction commonly using CTC or attention mechanisms.
- Rectification-based approaches address perspective distortion and curved text by transforming irregular text into a more regular representation before recognition.
- Other irregular-text methods use multi-direction encoding, two-dimensional attention, character segmentation, or specialized decoding losses.
- Prior work has introduced semantics through concepts, contextualized lexicons, or word-image embeddings for recognition and text retrieval tasks.
- The paper argues that recognition methods do not use text semantics well, while related semantic methods do not integrate semantics explicitly and effectively into the recognition pipeline.
3. Method
The proposed framework extends encoder-decoder recognition with explicit global semantic information, using it to supervise the encoder and guide decoder initialization. SE-ASTER integrates these components with rectification, visual feature extraction, semantic prediction, and attention-based decoding.
- 3.3.1 General Framework: The framework comprises an encoder with CNN and RNN components, a semantic module, a pre-trained language model, and an attention-based RNN decoder.The encoder extracts visual features, while the decoder generates recognition results.
- 3.3.1 General Framework: SEED adds a semantic module that predicts global semantic information from visual features and feeds it to the decoder as guidance.The semantic prediction is supervised by word embeddings from a pre-trained language model.
- 3.3.1 General Framework: SE-ASTER uses rectification, visual feature extraction, semantic prediction, and decoding as its four main modules.Rectification uses Thin-plate Splines after predicting control points; the encoder uses a ResNet-based CNN and Bidirectional LSTM.
- 3.3.2 Architecture of Semantics Enhanced ASTER: The semantic information is predicted from a flattened feature sequence using two linear functions with trainable weights, biases, and ReLU activation.The feature sequence serves both semantic prediction and decoder input; BiLSTM outputs perform better than the final hidden state because semantic prediction needs larger feature contexts.
- 3.3.2 Architecture of Semantics Enhanced ASTER: The decoder initializes its GRU states from transformed semantic information instead of zero states, combining global semantics with local visual features.It uses Bahdanau attention with a single-layer attentional GRU and a single-direction decoder.
- 3.4. Loss Function and Training Strategy: Training is end-to-end with recognition and semantic supervision, using a cosine embedding loss against FastText word embeddings and setting λ to 1.The predicted semantic information is supervised by the transcription label's pre-trained word embedding; direct predicted-semantic initialization supports pure end-to-end training.
4. Experiments
Experiments evaluate SEED through ablations, incomplete-character tests, framework integration, qualitative analyses, and benchmark comparisons. Results show stronger robustness on degraded text, with gains especially on IC15 and SVTP.
- Ablation study: Combining word-embedding supervision with decoder initialization produces the best ablation results.The combined strategy improves accuracy by 1.9% on IC13, 2.3% on SVTP, and 1.6% on IC15.
- Performance with inaccurate bounding boxes: On shrink datasets, ASTER accuracy declines 19.5% on IC13-sr and 12.8% on IC15-sr, whereas SE-ASTER achieves the best results.Word-embedding supervision alone still struggles with shrink images, while encoder guidance yields 16.5% and 13.0% declines.
- Framework generalization: Integrating SEED with SAR yields improvements of 4.2% on IC15, 1.3% on SVT, and 2.3% on SVTP over the baseline.SE-SAR is comparable with SAR on IC13, where low-quality images are scarce.
- Qualitative results and visualization: Predicted semantic information aligns with semantically related words and distinguishes similarly spelled words such as “house” and “horse.”Cosine similarity is computed against word embeddings from each image’s lexicon; higher values indicate more similar semantics.
- Comparison with state-of-the-art: In lexicon-free evaluation, the method achieves 2 best and 3 second-best results out of 6 with only word-level annotations.SE-ASTER improves over ASTER by 3.9% on IC15 and 2.9% on SVTP, and surpasses ScRN by 1.3% on IC15 and 0.6% on SVTP.
- Comparison with state-of-the-art: SE-ASTER improves over ASTER by 0.4% on IIIT5K and 4.1% on CUTE, while reaching 89.6% on SVT and 92.8% on IC13.It is slightly worse than ESIR on SVT by 0.6% and method [2] on IC13 by 1.6%.
5. Conclusion and Future Works
The paper proposes a semantics enhanced encoder-decoder framework that predicts global semantic information, uses it to initialize decoding, and improves recognition accuracy especially for low-quality images.
- SEED predicts global semantic information supervised by word embeddings from a pre-trained language model.
- Using predicted semantic information as decoder initialization improves recognition accuracy especially for low-quality images.
- Integrating ASTER into the framework achieves superior results on several standard benchmark datasets.