PaperlayerView exact revision on arXiv

Source-linked AI summary

NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition

Fenfen Sheng, Zhineng Chen, Bo Xu

arXiv:1806.00926v2cs.CV

TL;DR

Existing scene text recognizers rely on recurrence or convolution, creating limits in training parallelization or model complexity. NRTR replaces both with self-attention-based encoder-decoder processing and a modality-transform block for image-to-sequence conversion. It achieves state-of-the-art or highly competitive benchmark performance while training at least 8 times faster than the existing best recognizer.

  • Problem

    RNN-based recognizers limit computation parallelization, while CNN-based recognizers require greater complexity to learn distant dependencies.

    Source: §I · pp. 1–2

  • Method

    NRTR is a no-recurrence sequence-to-sequence recognizer using self-attention in its encoder and decoder plus a modality-transform block that converts 2D images to 1D sequences.

    Source: §I · pp. 1–2

  • Results

    NRTR achieves state-of-the-art or highly competitive performance on regular and irregular benchmarks and trains at least 8 times faster than the existing best recognizer.

    Source: §I · p. 2, 2) Speed · p. 5

  • Takeaways & Limitations

    NRTR demonstrates that a self-attention-only recognizer can provide competitive scene text recognition accuracy with greater computation parallelization and less complexity.

    Source: §I · p. 2

Abstract

from arXiv · show

Scene text recognition has attracted a great many researches due to its importance to various applications. Existing methods mainly adopt recurrence or convolution based networks. Though have obtained good performance, these methods still suffer from two limitations: slow training speed due to the internal recurrence of RNNs, and high complexity due to stacked convolutional layers for long-term feature extraction. This paper, for the first time, proposes a no-recurrence sequence-to-sequence text recognizer, named NRTR, that dispenses with recurrences and convolutions entirely. NRTR follows the encoder-decoder paradigm, where the encoder uses stacked self-attention to extract image features, and the decoder applies stacked self-attention to recognize texts based on encoder output. NRTR relies solely on self-attention mechanism thus could be trained with more parallelization and less complexity. Considering scene image has large variation in text and background, we further design a modality-transform block to effectively transform 2D input images to 1D sequences, combined with the encoder to extract more discriminative features. NRTR achieves state-of-the-art or highly competitive performance on both regular and irregular benchmarks, while requires only a small fraction of training time compared to the best model from the literature (at least 8 times faster).

I. INTRODUCTION

Scene text recognition remains challenging, while RNN- and CNN-based recognizers face efficiency and long-range dependency limitations. NRTR addresses these issues with a self-attention sequence-to-sequence design, a modality-transform block, and competitive benchmark performance.

  • Scene text recognition is difficult because images contain low-quality text, arbitrary orientations, cluttered backgrounds, and complex deformations.Source: §I · p. 1
  • RNN recognizers capture contextual character dependencies but limit computation parallelization and can suffer gradient vanishing or exploding.Source: §I · p. 1
  • CNN recognizers enable parallel computation but require additional convolutional layers to model distant dependencies, increasing complexity.Source: §I · p. 1
  • NRTR is a no-recurrence sequence-to-sequence recognizer that dispenses with recurrences and convolutions and relies solely on self-attention.Source: §I · p. 1
  • NRTR uses stacked self-attention in its encoder and decoder to connect input and output positions globally while enabling greater computation parallelization.Source: §I · p. 2
  • The modality-transform block converts 2D scene images into 1D sequences before encoding to extract more discriminative features.Source: §I · p. 2
  • NRTR achieves state-of-the-art or highly competitive performance on regular and irregular benchmarks, with training at least 8 times faster than the existing best recognizer.Source: §I · p. 2

II. RELATED WORK

Related scene text recognizers formulate recognition as sequence prediction, using recurrent or convolutional architectures to process image and character sequences.

B. Our Method Versus Some Related Works

NRTR adapts self-attention from Transformer to scene text recognition through a dedicated encoder-decoder architecture and image-specific preprocessing.

B. Modality-Transform Block

NRTR’s modality-transform block preprocesses 2D images into encoder-ready 1D sequences, while positional encoding preserves sequence positions without recurrence.

B. Modality-Transform Block

C. Encoder

The encoder stacks identical blocks built around multi-head self-attention and position-wise fully connected networks to transform image sequences into hidden representations.

C. Encoder

  • The encoder contains N_e connected identical blocks, each with multi-head scaled dot-product attention and a position-wise fully connected network.Source: C. Encoder · p. 3
  • Multi-head attention jointly attends to different representation subspaces and positions through parallel scaled dot-product attention operations.Source: C. Encoder · p. 3
  • Multi-head attention projects queries, keys, and values, concatenates the parallel head outputs, and applies a linear layer.Source: C. Encoder · p. 3
  • The position-wise fully connected network applies two linear transformations with a ReLU activation between them.Source: C. Encoder · p. 3
  • Layer normalization and residual connections are applied around each encoder sub-layer for effective training.Source: C. Encoder · p. 3

D. Decoder

The decoder converts character labels into positional vectors, processes them with stacked self-attention blocks, and predicts character probabilities from encoder outputs.

D. Decoder

  • The decoder generates text sequences from encoder outputs and input labels, embedding each character into a d_model-dimensional vector with positional encoding.Source: D. Decoder · pp. 3–4
  • The decoder contains N_d connected identical blocks based on multi-head attention and position-wise fully connected networks.Source: D. Decoder · p. 4
  • Masked multi-head attention ensures prediction at position j depends only on known outputs before j.Source: D. Decoder · p. 4
  • A linear projection and softmax transform decoder outputs into probabilities for character classes.Source: D. Decoder · p. 4

IV. EXPERIMENT

The experiments evaluate NRTR on standard regular and irregular scene-text benchmarks using a fixed training and inference setup, with results summarized in Table I.

IV. EXPERIMENT

  • The evaluation covers regular benchmarks IIIT5K, SVT, ICDAR2003, and ICDAR2013, plus irregular datasets SVT-P, CUTE80, and ICDAR2015.Source: A. Benchmark Datasets · p. 4
  • NRTR is trained on Synth90k and evaluated on standard benchmarks without finetuning, with input heights fixed at 32 and proportionally scaled widths.Source: B. Implementation Details · p. 4
  • The output vocabulary contains 38 classes: 26 lowercase letters, 10 digits, one space, and one end-of-sequence token.Source: B. Implementation Details · p. 4
  • Table I explores encoder, decoder, and modality-transform block configurations on lexicon-free benchmarks.Source: Table I · p. 4, §IV · p. 4
  • Training batches group samples by approximate image widths, and optimization uses Adam with β1 = 0.9, β2 = 0.98, and ϵ = 10^-9.Source: B. Implementation Details · p. 4

C. Ablation Study

The ablation study examines encoder depth, decoder depth, and fully connected dimension under a shared lexicon-free training and evaluation strategy.

C. Ablation Study

2) Exploration of the modality-transform block:

The modality-transform block converts scene images into sequences before encoding, with architecture choices affecting recognition performance. Two convolutional layers are selected for the final NRTR configuration, while CNNLSTM helps only in the base model.

Modality-transform architectures

  • More convolutional layers reduced performance, including the seven-layer convnet used in CRNN and RARE.The authors attribute this to resolution subsampling outweighing higher-level semantic gains.Source: 2) Exploration of the… · p. 5
  • The selected block uses two convolutional layers combined with the encoder to extract more discriminative features.The encoder’s feature-extraction ability motivated limiting the block to two convolutional layers.Source: 2) Exploration of the… · p. 5
  • CNNLSTM improved the base model’s accuracy but slightly reduced accuracy in the larger model.The authors conjecture that excessive encoder components create redundant image-information extraction when combined with CNNLSTM.Source: 2) Exploration of the… · p. 5

Final configuration

  • The final NRTR sets Ne = 12, Nd = 6, dff = 4096 and uses a two-convolution modality-transform block.This configuration was established after the preceding architecture analysis and was also used for quantitative comparisons.Source: D. Comparisons with the… · p. 5

Benchmark results

  • NRTR surpasses previous recognizers across regular benchmarks in lexicon-free cases and generally improves accuracy in lexicon-based cases.Reported gains include 0.9% on IIIT5K with a 1k lexicon, 1.5% on SVT, and 0.2% on IC03 with the Full lexicon, alongside a 0.3% IIIT5K decline with 50-word lexicons.Source: 1) Accuracy · p. 5
  • NRTR remains highly tolerant of irregular texts without special operations, despite comparison models being designed specifically for irregular benchmarks.This result is presented as evidence of strong text-feature extraction ability.Source: D. Comparisons with the… · p. 5

2) Speed:

NRTR substantially reduces training time while maintaining competitive recognition across regular and irregular benchmarks. Its qualitative failures remain concentrated in occlusion, similar-looking characters, and severe curvature.

Training speed

  • 5.0 hours per epoch makes NRTR at least 8 times faster to train than the existing best recognizer.The best previous method requires 41 hours per epoch on a P40, while NRTR requires 5.0 hours per epoch.Source: 2) Speed · p. 5

Inference speed

  • 0.03s per image is NRTR’s approximate inference time, compared with 0.11s for and 0.2s for.These inference times are reported alongside the training-speed comparison.Source: 2) Speed · p. 5

Failure cases

  • Incorrect examples involve severe occlusion, confusion between similar-looking characters, and text orientations curved nearly ninety degrees from horizontal.These failure types are identified as future research directions.Source: 3) Visualization · p. 5

Related papers

Loading 1806.00926v2…