PaperlayerView exact revision on arXiv

Source-linked AI summary

SCATTER: Selective Context Attentional Scene Text Recognizer

Ron Litman, Oron Anschel, Shahar Tsiper, Roee Litman, Shai Mazor, R. Manmatha

arXiv:2003.11288v1cs.CV

TL;DR

Scene Text Recognition remains challenging for text with arbitrary shapes and difficult visual conditions. SCATTER uses stacked processing with intermediate supervision and selective two-step attention, achieving state-of-the-art recognition accuracy on irregular text benchmarks.

  • Problem

    Scene Text Recognition must decode natural-scene text under difficult visual conditions, including arbitrarily shaped text.

  • Method

    SCATTER combines stacked repetitive processing and intermediate supervision with a selective decoder using two-step attention over CNN visual and BiLSTM contextual features.

  • Results

    SCATTER with five blocks outperforms Mask TextSpotter on irregular benchmarks by 3.7% on average.

  • Takeaways & Limitations

    Intermediate selective decoders increasingly refine text predictions and support stable training of deep BiLSTM encoders.

  • Takeaways & Limitations

    The method assumes a cropped natural-image text input and outputs the recognized string within that crop.

Abstract

from arXiv · show

Scene Text Recognition (STR), the task of recognizing text against complex image backgrounds, is an active area of research. Current state-of-the-art (SOTA) methods still struggle to recognize text written in arbitrary shapes. In this paper, we introduce a novel architecture for STR, named Selective Context ATtentional Text Recognizer (SCATTER). SCATTER utilizes a stacked block architecture with intermediate supervision during training, that paves the way to successfully train a deep BiLSTM encoder, thus improving the encoding of contextual dependencies. Decoding is done using a two-step 1D attention mechanism. The first attention step re-weights visual features from a CNN backbone together with contextual features computed by a BiLSTM layer. The second attention step, similar to previous papers, treats the features as a sequence and attends to the intra-sequence relationships. Experiments show that the proposed approach surpasses SOTA performance on irregular text recognition benchmarks by 3.7\% on average.

1. Introduction

Scene Text Recognition decodes cropped natural-scene text despite challenging visual conditions and arbitrary shapes. SCATTER addresses this problem with stacked processing, intermediate supervision, and selective two-step attention over visual and contextual features.

  • 1. Introduction: Natural-scene text is difficult because backgrounds and imaging conditions produce blur, distortion, low resolution, uneven illumination, and low contrast.Font variation in type and size adds further difficulty.
  • 1. Introduction: SCATTER targets recognition of cropped natural-scene text, including arbitrarily shaped text such as curved text.The task outputs the recognized text string from a cropped image.
  • 1. Introduction: Traditional character-by-character methods do not model sequential or contextual dependencies between characters.Modern sequence-based approaches instead formulate recognition as sequence prediction using CTC or attention mechanisms.
  • 1. Introduction: Training with additional blocks improves the accuracy of intermediate decoders, while deeper training can support inference using only earlier decoders.Figure 2 compares average accuracy across seven regular and irregular benchmarks at intermediate decoding steps.
  • 1. Introduction: SCATTER uses stacked repetitive processing with intermediate supervision to train a deep BiLSTM encoder and selective decoding over CNN visual and BiLSTM contextual features.Its two-step 1D attention first selects visual and contextual features, then models intra-sequence relationships.

2. Related Work

Related STR research moved from character-based recognition toward sequence modeling and methods specialized for irregular text. SCATTER distinguishes itself by applying stacked repetitive processing with intermediate selective decoders for text recognition.

  • 2. Related Work: Earlier STR methods segmented and classified individual characters, while sequence-based methods modeled words as variable-length sequences without explicit character separation.BiLSTM-based sequence extraction improved end-to-end trainable recognition without character-level annotations.
  • 2. Related Work: Recent work focused increasingly on irregular text through rectification, character-aware segmentation, 2D attention, self-attention, and transformer-based architectures.These approaches target the challenge of recognizing arbitrarily shaped text.
  • 2. Related Work: SCATTER differs from prior approaches by being the first described here to use a stacked block architecture for text recognition.Its intermediate selective decoders supervise repetitive processing that increasingly refines text predictions.

3. Methodology

SCATTER combines stacked contextual-refinement blocks, intermediate supervision, and a two-step selective decoder for scene-text recognition. Its architecture repeatedly enriches visual features with BiLSTM context before attention-based character decoding.

  • 3. Methodology: SCATTER comprises transformation, feature extraction, visual feature refinement, and stacked selective-contextual refinement components.The architecture uses visual feature supervision and repeated contextual processing before final decoding.
  • 3.3. Visual Feature Refinement: During training, intermediate CTC and attention decoders provide supervision for visual columns and refinement blocks, while intermediate decoders are removed at inference.The final inference path retains BiLSTM processing and the final selective decoder; intermediate decoders serve training supervision only.
  • 3.4. Selective-Contextual Refinement Block: Each refinement block applies a two-layer BiLSTM to visual features, concatenates contextual and visual representations, and passes the result to subsequent blocks.The visual feature map remains fixed while contextual features are updated across stacked blocks.
  • 3.4.1 Selective-Decoder: The selective decoder first reweights concatenated visual-contextual features with 1D attention, then applies a separate attention decoder to predict characters sequentially.The first step produces attentional features, and the second decoder operates on them at each output time step.
  • 3.5. Training Losses: Training minimizes a weighted combination of the CTC loss and attention losses from all Selective-Contextual Refinement blocks.The loss weights balance the different supervision signals, with λCTC empirically set to 0.1 and λj to 1.0.

4. Experiments

SCATTER is evaluated on seven real-world scene-text benchmarks using synthetic training data, with comparisons covering accuracy, ablations, computational cost, and failure cases. It achieves its strongest gains on irregular text while remaining competitive on regular text.

  • Datasets: The experiments train SCATTER on MJSynth, SynthText, and SynthAdd, then evaluate it on four regular and three irregular real-world datasets.The training datasets contain 9 million, 5.5 million, and 1.2 million word-box images, respectively.
  • Comparison to State-of-the-art: 3.7 percentage points: five-block SCATTER outperforms Mask TextSpotter on average across IC15, SVTP, and CUTE irregular benchmarks.The gains are +4.0 pp on IC15 and +3.3 pp on SVTP; SCATTER is second best on CUTE.
  • Comparison to State-of-the-art: SCATTER achieves the highest recognition score on 4 of 7 benchmarks and the second-highest score on 2 others.The model is reported as a top performer across both regular and irregular benchmark groups.
  • Computational Costs: A five-block model has a 32.9 ms forward pass, adding 12.8 ms over the single-block configuration.Each intermediate contextual refinement block adds 3.2 ms during inference.
  • Computational Costs: Training with more blocks and pruning for inference improves accuracy by 0.4 pp on regular data and 1.3 pp on irregular data under the same computational budget.The reported setup trains with five blocks and prunes to a single block for inference.

5. Ablation Experiments

The ablations show that intermediate supervision and selective decoding improve SCATTER, while stacked supervision enables deeper BiLSTM encoders and progressively refined predictions. Accuracy improves through 10 layers but declines beyond that point, and oracle decoder selection indicates additional potential.

  • Ablations: +2.9 pp irregular-text accuracy is achieved by increasing intermediate supervisions in the four-layer SCATTER architecture.The corresponding increase is +0.7 pp on regular text, with accuracy rising monotonically as supervisions increase from one to three.
  • Deep BiLSTM encoder: Accuracy improves monotonically through 10 BiLSTM layers in SCATTER for both regular and irregular text.This contrasts with the baseline architecture, where accuracy decreases beyond two BiLSTM layers.
  • Deep BiLSTM encoder: Accuracy decreases slightly beyond 10 BiLSTM layers, by -0.4 pp on regular text and -0.5 pp on irregular text.The authors attribute this setting to more challenging training procedures and suggest alternative approaches may be needed for convergence.
  • Intermediate predictions: Repeated processing with intermediate supervision increasingly refines text predictions across intermediate decoders.Table 3 provides examples, while Table 4 reports decoder-specific performance and an oracle that selects a correct prediction when available.

6. Conclusions and Future Work

SCATTER combines stacked processing, intermediate supervision, and a selective decoder to achieve SOTA recognition accuracy and support deep BiLSTM encoders. The authors identify pruning and decoder-selection strategies as future directions.

  • Conclusions: SCATTER achieves SOTA recognition accuracy and enables stable, more robust training of STR networks with deep BiLSTM encoders.The architecture adds intermediate supervisions along network layers and uses a novel selective decoder.
  • Conclusions: The selective decoder attends to visual and contextual features through a two-step attention mechanism.The first step selects visual and contextual features, while the second models intra-sequence relations.
  • Future implications: Repeated processing with intermediate selective decoders increasingly refines text predictions, and the stacked-decoder idea may extend to other attention approaches.The authors state that the novelty is not limited to their particular attention formulation.
  • Future work: Training deeper networks and pruning their final decoders and layers is proposed as a way to improve performance under computational-budget constraints.The authors also propose developing an optimal strategy for selecting among decoder predictions for each image.

A. Regular Vs Irregular Text

Scene-text datasets distinguish irregular text, which may be arbitrarily shaped, from regular text, whose characters are nearly horizontally aligned.

  • A. Regular Vs Irregular Text: Irregular text can be arbitrarily shaped, whereas regular text has a nearly horizontally aligned character sequence.The paper uses curved text as an example of irregular text and contrasts the categories with examples in Figure 6.

B. Network Pruning - Compute Constraint

The pruning analysis compares intermediate-decoder accuracy across stacking arrangements and supports training deeper networks before pruning.

  • B. Network Pruning - Compute Constraint: Training a deep network with more blocks and then pruning is generally preferable to training a shallow architecture directly.Table 5 provides the exact results for stacking arrangements from one through five blocks, extending Figure 2.

C. Examples of Intermediate Predictions

Intermediate decoders can succeed when the final decoder fails, while earlier decoders sometimes fail before the final decoder succeeds. This disagreement suggests selecting or combining decoder outputs to improve recognition.

  • C. Examples of Intermediate Predictions: Intermediate and final decoders do not always agree: earlier decoders can fail while the final decoder succeeds, and intermediate decoders can also succeed when the final decoder fails.The examples cover both regular and irregular text images.
  • C. Examples of Intermediate Predictions: The complementary errors motivate selection, voting, or ensemble methods over the available selective-decoder outputs.Such methods could choose a correct prediction when different decoding stages disagree.

D. Stable Training of a Deep BiLSTM Encoder

SCATTER enables stacking additional BiLSTM layers despite the baseline’s accuracy degradation beyond two layers. Its intermediate decoding stages are evaluated with weighted averages across regular and irregular datasets.

  • D. Stable Training of a Deep BiLSTM Encoder: The baseline loses accuracy when more than two BiLSTM layers are added to its encoder.This reproduces the reduction reported in the earlier experiment.
  • D. Stable Training of a Deep BiLSTM Encoder: Table 5 reports average test accuracy at intermediate decoding stages across different training network depths.The regular and irregular columns use size-weighted averages over their respective datasets.
  • D. Stable Training of a Deep BiLSTM Encoder: SCATTER allows stacking more BiLSTM layers, ultimately increasing final recognition performance.The baseline experiment shows reduced accuracy beyond two BiLSTM layers, whereas SCATTER supports deeper encoders.
  • D. Stable Training of a Deep BiLSTM Encoder: Examples in Table 6 indicate that intermediate-decoder outputs could support selection, voting, or ensemble improvements.The table includes examples from both regular and irregular text datasets.
  • D. Stable Training of a Deep BiLSTM Encoder: Table 7 compares recognition accuracy across BiLSTM-layer counts for a retrained Baek et al. 2019 baseline and SCATTER.Regular Text and Irregular Text results are weighted averages by dataset size.
Loading 2003.11288v1…