PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Accurate Scene Text Recognition with Semantic Reasoning Networks

Deli Yu, Xuan Li, Chengquan Zhang, Junyu Han, Jingtuo Liu, Errui Ding

arXiv:2003.12294v1cs.CV

TL;DR

Scene text recognition has underused semantic information, while RNN-like approaches decode sequentially and transmit context one way. SRN addresses this with global semantic reasoning through multi-way parallel transmission, achieving state-of-the-art results across seven benchmarks spanning regular, irregular, and non-Latin long text.

  • Problem

    RNN-like scene text recognition methods reason sequentially from previous labels or predictions, limiting semantic reasoning and inference efficiency.

  • Method

    SRN is an end-to-end trainable framework that combines PVAM, GSRM, and VSFD to fuse aligned visual and semantic features for character prediction.

  • Results

    SRN achieves state-of-the-art results on almost seven public benchmarks covering regular text, irregular text, and non-Latin long text.

  • Takeaways & Limitations

    Global semantic reasoning through multi-way parallel transmission outperforms one-way semantic reasoning by about 1% accuracy on most benchmarks.

  • Takeaways & Limitations

    SRN may fail when visual features are misaligned or missed, or when both visual and semantic context fail for special fonts, low-quality images, or rare words.

Abstract

from arXiv · show

Scene text image contains two levels of contents: visual texture and semantic information. Although the previous scene text recognition methods have made great progress over the past few years, the research on mining semantic information to assist text recognition attracts less attention, only RNN-like structures are explored to implicitly model semantic information. However, we observe that RNN based methods have some obvious shortcomings, such as time-dependent decoding manner and one-way serial transmission of semantic context, which greatly limit the help of semantic information and the computation efficiency. To mitigate these limitations, we propose a novel end-to-end trainable framework named semantic reasoning network (SRN) for accurate scene text recognition, where a global semantic reasoning module (GSRM) is introduced to capture global semantic context through multi-way parallel transmission. The state-of-the-art results on 7 public benchmarks, including regular text, irregular text and non-Latin long text, verify the effectiveness and robustness of the proposed method. In addition, the speed of SRN has significant advantages over the RNN based methods, demonstrating its value in practical use.

1. Introduction

Scene text recognition must combine visual features with semantic context because difficult characters can be ambiguous visually. The paper addresses limits of serial semantic transmission with SRN, whose GSRM reasons over global context in parallel.

  • Scene text varies substantially in color, font, spatial layout, and background, making recognition in the wild challenging.
  • Visual features alone can make individual characters difficult to distinguish, whereas whole-word semantic context can help infer the correct result.
  • One-way serial semantic transmission sees limited context, can accumulate errors from earlier wrong decodings, and is difficult to parallelize efficiently.
  • GSRM uses multi-way parallel transmission to perceive semantic information across all characters while limiting the impact of an individual character error.
  • SRN combines GSRM with PVAM and VSFD in an end-to-end trainable framework for scene text recognition.
  • SRN achieves state-of-the-art performance on benchmarks covering regular, irregular, and non-Latin long text.

2. Related Work

Prior scene text recognition work includes semantic context-free visual methods and semantic context-aware methods, while context modeling research balances parallel efficiency against global relation modeling. SRN targets these gaps with a pipeline centered on global semantic reasoning.

  • Related methods are categorized as semantic context-free or semantic context-aware according to whether semantic information is used.
  • Semantic context-free methods treat recognition as visual classification without explicitly using semantic information, exemplified by CRNN with a CTC decoder.
  • Semantic context-aware methods typically transmit semantic information one way, using previous decoding information to guide later visual attention.
  • Existing visual-feature improvements do not fully exploit semantic context information, which is the focus of this paper.
  • RNNs capture sequence dependencies but hinder parallel computation, whereas CNN-based encoders parallelize computation but cannot flexibly capture global relations.

3. Approach

SRN combines parallel visual attention with global semantic reasoning and visual-semantic fusion to recognize scene text without recursive decoding. Its modules align character features, infer semantic context across positions, and predict characters in parallel.

  • Overall framework: SRN processes backbone features through PVAM, GSRM, and VSFD to produce character predictions from aligned visual and semantic information.The backbone extracts 2D features; PVAM generates aligned 1-D character features, GSRM derives semantic features, and VSFD fuses both domains.
  • Parallel Visual Attention Module: PVAM replaces the time-dependent recurrent query with character reading order, enabling all aligned visual features to be computed in parallel.Reading-order embeddings serve as queries instead of the previous hidden state, eliminating time-dependent attention terms.
  • Parallel Visual Attention Module: PVAM attention maps focus on the visual areas corresponding to individual characters, supporting the intended feature-alignment function.The module aligns visual information to each output character rather than relying on sequential attention updates.
  • Global Semantic Reasoning Module: GSRM uses time-independent approximate embeddings and transformer-based semantic reasoning to combine context from preceding and following characters.Its visual-to-semantic embedding block generates approximate character embeddings, while the semantic reasoning block outputs semantic features for every step.
  • Visual-Semantic Fusion Decoder: VSFD learns a gate that balances aligned visual features and semantic features before predicting the final character sequence non-recursively.The fused feature is formed from a trainable mixture of the visual feature g_t and semantic feature s_t.

4.1. Datasets

The evaluation uses six Latin scene-text benchmarks spanning regular, noisy, perspectively distorted, and curved text.

  • Six Latin scene-text benchmarks are used to evaluate the method.
  • ICDAR 2013 contains 1,095 testing images after filtering non-alphanumeric or shorter-than-three-character images.
  • ICDAR 2015 uses 1,811 test images after excluding extremely distorted images.
  • IIIT5k comprises 3,000 testing images collected from websites.
  • SVT contains 647 images, many corrupted by noise, blur, and low resolution.
  • SVTP contains 639 test images with many perspectively distorted samples, while CUTE contains 288 images for curved text recognition.

4.2. Implementation Details

The implementation uses fixed-dimensional transformer-based modules, augmented 64 × 256 inputs, synthetic-only training, and end-to-end optimization.

  • PVAM and GSRM embeddings are 512-dimensional, and GSRM contains four stacked transformer units with eight heads and 512 hidden units.
  • Training images are resized, padded to 64 × 256, and randomly augmented with geometric and image-processing transformations.
  • The model predicts 37 classes, including digits, lowercase letters, and EOS, with a maximum output length of 25.
  • SRN is trained only on Synth90K and SynthText, initialized from ImageNet-pretrained ResNet50, and trained with batch size 256.
  • Training uses warming-up without GSRM followed by joint end-to-end training until convergence on eight NVIDIA P40 GPUs.

4.3. Ablation Study

Ablations examine visual encoding, semantic reasoning, fusion, and qualitative corrections, showing benefits from transformer context, GSRM, and gated fusion while identifying failure boundaries.

  • Backbone and PVAM: Transformer units improve backbone accuracy by at least 3% across all benchmarks, while character reading order helps most public datasets, especially CUTE.
  • GSRM: Four-GSRM is adopted because its performance is similar to six-GSRM while preserving controllable computation.
  • Semantic reasoning strategy: Global semantic reasoning outperforms forward or backward one-way reasoning by about 1% accuracy on most benchmarks.
  • Feature fusion strategy: Alternative add, concatenate, and dot fusion operations degrade performance, motivating the default gated-unit VSFD.
  • Right and failure cases: GSRM corrects visually confusable characters such as “r” versus “c” in “Vernon” using global semantic context.
  • Right and failure cases: SRN cannot insert or remove characters, and may fail when visual and semantic context both fail, especially for special fonts, low quality, or rare words.

4.4. Comparisons with State-of-the-Arts

Across six public datasets, SRN performs strongly against state-of-the-art methods, including on distorted text without dedicated rectification modules.

  • SRN achieves superior performance across six public datasets compared with state-of-the-art approaches without lexicons.
  • SRN obtains increases of 4.0% on IC15 and 2.8% on SVTP compared with state-of-the-art methods using rectification modules.
  • SRN achieves comparable results on CUTE despite not using special measures for irregular words.

4.5. Results on non-Latin Long Text

On non-Latin long text, SRN improves recognition over CTC, attention-based methods, and SCCM, while GSRM helps resolve visually similar characters through semantic context.

  • 6.8% and 8.4% boosts in TRW-T are achieved over CTC- and attention-based methods by the proposed approach without GSRM.
  • 4.9% improvement from GSRM over SRN without GSRM exceeds the 4.7% improvement obtained by SCCM’s LM model.
  • GSRM’s improvement increases from 4.9% on TRW-T to 6.8% on TRW-L, supporting its value for long-text recognition.
  • Semantic information helps SRN distinguish visually similar characters, such as correctly inferring “责” from the phrase “素 材”.

4.6. Inference Speed

SRN’s parallel design provides substantial inference-speed advantages over 1D- and 2D-Attention methods on both short and long text, while removing GSRM retains CTC-like efficiency.

  • Speed is evaluated on IC15 and TRW-L using the same backbone, hardware, and three runs per method.
  • SRN with GSRM is 1.7 times and 1.8 times faster than 1D- and 2D-Attention methods on IC15.
  • 2.0 times and 2.2 times acceleration over 1D- and 2D-Attention methods is achieved on the long-text dataset TRW-L.
  • SRN without GSRM has computational efficiency similar to CTC-based methods because of its parallelism and simplicity.

5. Conclusion

The conclusion presents SRN as an end-to-end framework that models semantic context with GSRM and achieves strong benchmark performance with parallel execution.

  • SRN uses GSRM to model first-order and higher-order semantic relations among Latin or non-Latin characters.
  • SRN combines GSRM with a backbone network, parallel visual attention module, and fusion decoder module.
  • SRN achieves state-of-the-art results on almost seven public benchmarks spanning regular, irregular, and non-Latin long text.
  • Because all SRN modules are time independent, the framework can run in parallel and is presented as more practical than other semantic modeling methods.
Loading 2003.12294v1…