PaperlayerView exact revision on arXiv

Source-linked AI summary

NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition

Fenfen Sheng, Zhineng Chen, Bo Xu

arXiv:1806.00926v2cs.CV

TL;DR

Existing scene text recognizers rely on recurrence or convolution, creating limits in training parallelization or model complexity. NRTR replaces both with self-attention-based encoder-decoder processing and a modality-transform block for image-to-sequence conversion. It achieves state-of-the-art or highly competitive benchmark performance while training at least 8 times faster than the existing best recognizer.

  • Problem

    RNN-based recognizers limit computation parallelization, while CNN-based recognizers require greater complexity to learn distant dependencies.

  • Method

    NRTR is a no-recurrence sequence-to-sequence recognizer using self-attention in its encoder and decoder plus a modality-transform block that converts 2D images to 1D sequences.

  • Results

    NRTR achieves state-of-the-art or highly competitive performance on regular and irregular benchmarks and trains at least 8 times faster than the existing best recognizer.

  • Takeaways & Limitations

    NRTR demonstrates that a self-attention-only recognizer can provide competitive scene text recognition accuracy with greater computation parallelization and less complexity.

Abstract

from arXiv · show

Scene text recognition has attracted a great many researches due to its importance to various applications. Existing methods mainly adopt recurrence or convolution based networks. Though have obtained good performance, these methods still suffer from two limitations: slow training speed due to the internal recurrence of RNNs, and high complexity due to stacked convolutional layers for long-term feature extraction. This paper, for the first time, proposes a no-recurrence sequence-to-sequence text recognizer, named NRTR, that dispenses with recurrences and convolutions entirely. NRTR follows the encoder-decoder paradigm, where the encoder uses stacked self-attention to extract image features, and the decoder applies stacked self-attention to recognize texts based on encoder output. NRTR relies solely on self-attention mechanism thus could be trained with more parallelization and less complexity. Considering scene image has large variation in text and background, we further design a modality-transform block to effectively transform 2D input images to 1D sequences, combined with the encoder to extract more discriminative features. NRTR achieves state-of-the-art or highly competitive performance on both regular and irregular benchmarks, while requires only a small fraction of training time compared to the best model from the literature (at least 8 times faster).

I. INTRODUCTION

Scene text recognition remains challenging, while RNN- and CNN-based recognizers face efficiency and long-range dependency limitations. NRTR addresses these issues with a self-attention sequence-to-sequence design, a modality-transform block, and competitive benchmark performance.

  • Scene text recognition is difficult because images contain low-quality text, arbitrary orientations, cluttered backgrounds, and complex deformations.
  • RNN recognizers capture contextual character dependencies but limit computation parallelization and can suffer gradient vanishing or exploding.
  • CNN recognizers enable parallel computation but require additional convolutional layers to model distant dependencies, increasing complexity.
  • NRTR is a no-recurrence sequence-to-sequence recognizer that dispenses with recurrences and convolutions and relies solely on self-attention.
  • NRTR uses stacked self-attention in its encoder and decoder to connect input and output positions globally while enabling greater computation parallelization.
  • The modality-transform block converts 2D scene images into 1D sequences before encoding to extract more discriminative features.
  • NRTR achieves state-of-the-art or highly competitive performance on regular and irregular benchmarks, with training at least 8 times faster than the existing best recognizer.

II. RELATED WORK

Related scene text recognizers formulate recognition as sequence prediction, using recurrent or convolutional architectures to process image and character sequences.

  • Modern text recognition methods commonly represent input images and output texts as patch and character sequences within a sequence-to-sequence paradigm.
  • CNN-RNN systems learn spatial dependencies and use CTC to translate per-slice predictions into label sequences.
  • Attention-based recurrent networks decode feature sequences and predict labels recurrently.
  • Stacked CNNs have been used instead of RNNs to pursue greater computational parallelism.

B. Our Method Versus Some Related Works

NRTR adapts self-attention from Transformer to scene text recognition through a dedicated encoder-decoder architecture and image-specific preprocessing.

  • Transformer is the closest related work, but it targets English-to-French translation and does not directly read text in natural images.
  • NRTR uses self-attention throughout a scene-text-specific encoder and decoder rather than recurrences or convolutions.
  • The architecture contains three sub-networks: an encoder, a decoder, and a modality-transform block used for preprocessing.
  • Self-attention connects all positions with a constant number of operations and supports parallelization.

B. Modality-Transform Block

NRTR’s modality-transform block preprocesses 2D images into encoder-ready 1D sequences, while positional encoding preserves sequence positions without recurrence.

  • B. Modality-Transform Block: The modality-transform block uses convolutional layers with stride 2 and progressively doubles channel numbers.
  • B. Modality-Transform Block: At each layer, the product of image height and channel number remains equal to d_model, the encoder-decoder feature dimension.
  • B. Modality-Transform Block: After the final layer, concatenation reshapes features into an input sequence whose elements each have d_model dimensions.
  • B. Modality-Transform Block: Because NRTR has no recurrences, positional encoding indicates each position in the image sequence.
  • B. Modality-Transform Block: The final encoder input sequence is formed by adding positional encoding to the transformed image sequence.

C. Encoder

The encoder stacks identical blocks built around multi-head self-attention and position-wise fully connected networks to transform image sequences into hidden representations.

  • C. Encoder: The encoder contains N_e connected identical blocks, each with multi-head scaled dot-product attention and a position-wise fully connected network.
  • C. Encoder: Multi-head attention jointly attends to different representation subspaces and positions through parallel scaled dot-product attention operations.
  • C. Encoder: Multi-head attention projects queries, keys, and values, concatenates the parallel head outputs, and applies a linear layer.
  • C. Encoder: The position-wise fully connected network applies two linear transformations with a ReLU activation between them.
  • C. Encoder: Layer normalization and residual connections are applied around each encoder sub-layer for effective training.

D. Decoder

The decoder converts character labels into positional vectors, processes them with stacked self-attention blocks, and predicts character probabilities from encoder outputs.

  • D. Decoder: The decoder generates text sequences from encoder outputs and input labels, embedding each character into a d_model-dimensional vector with positional encoding.
  • D. Decoder: The decoder contains N_d connected identical blocks based on multi-head attention and position-wise fully connected networks.
  • D. Decoder: Masked multi-head attention ensures prediction at position j depends only on known outputs before j.
  • D. Decoder: A linear projection and softmax transform decoder outputs into probabilities for character classes.

IV. EXPERIMENT

The experiments evaluate NRTR on standard regular and irregular scene-text benchmarks using a fixed training and inference setup, with results summarized in Table I.

  • IV. EXPERIMENT: The evaluation covers regular benchmarks IIIT5K, SVT, ICDAR2003, and ICDAR2013, plus irregular datasets SVT-P, CUTE80, and ICDAR2015.
  • IV. EXPERIMENT: NRTR is trained on Synth90k and evaluated on standard benchmarks without finetuning, with input heights fixed at 32 and proportionally scaled widths.
  • IV. EXPERIMENT: The output vocabulary contains 38 classes: 26 lowercase letters, 10 digits, one space, and one end-of-sequence token.
  • IV. EXPERIMENT: Table I explores encoder, decoder, and modality-transform block configurations on lexicon-free benchmarks.
  • IV. EXPERIMENT: Training batches group samples by approximate image widths, and optimization uses Adam with β1 = 0.9, β2 = 0.98, and ϵ = 10^-9.

C. Ablation Study

The ablation study examines encoder depth, decoder depth, and fully connected dimension under a shared lexicon-free training and evaluation strategy.

  • C. Ablation Study: All ablation experiments use the same training strategy and are evaluated under the lexicon-free setting.
  • C. Ablation Study: The study varies encoder-block number N_e, decoder-block number N_d, and fully connected inner dimension d_ff.
  • C. Ablation Study: With identical total block counts, configurations with more encoder blocks achieve better accuracy than alternatives with more decoder blocks.
  • C. Ablation Study: The baseline is a 6enc6dec model, while deeper models obtain higher accuracy when additional blocks are added.

2) Exploration of the modality-transform block:

The modality-transform block converts scene images into sequences before encoding, with architecture choices affecting recognition performance. Two convolutional layers are selected for the final NRTR configuration, while CNNLSTM helps only in the base model.

  • Modality-transform architectures: More convolutional layers reduced performance, including the seven-layer convnet used in CRNN and RARE.The authors attribute this to resolution subsampling outweighing higher-level semantic gains.
  • Modality-transform architectures: The selected block uses two convolutional layers combined with the encoder to extract more discriminative features.The encoder’s feature-extraction ability motivated limiting the block to two convolutional layers.
  • Modality-transform architectures: CNNLSTM improved the base model’s accuracy but slightly reduced accuracy in the larger model.The authors conjecture that excessive encoder components create redundant image-information extraction when combined with CNNLSTM.
  • Final configuration: The final NRTR sets Ne = 12, Nd = 6, dff = 4096 and uses a two-convolution modality-transform block.This configuration was established after the preceding architecture analysis and was also used for quantitative comparisons.
  • Benchmark results: NRTR surpasses previous recognizers across regular benchmarks in lexicon-free cases and generally improves accuracy in lexicon-based cases.Reported gains include 0.9% on IIIT5K with a 1k lexicon, 1.5% on SVT, and 0.2% on IC03 with the Full lexicon, alongside a 0.3% IIIT5K decline with 50-word lexicons.
  • Benchmark results: NRTR remains highly tolerant of irregular texts without special operations, despite comparison models being designed specifically for irregular benchmarks.This result is presented as evidence of strong text-feature extraction ability.

2) Speed:

NRTR substantially reduces training time while maintaining competitive recognition across regular and irregular benchmarks. Its qualitative failures remain concentrated in occlusion, similar-looking characters, and severe curvature.

  • Training speed: 5.0 hours per epoch makes NRTR at least 8 times faster to train than the existing best recognizer.The best previous method requires 41 hours per epoch on a P40, while NRTR requires 5.0 hours per epoch.
  • Inference speed: 0.03s per image is NRTR’s approximate inference time, compared with 0.11s for and 0.2s for.These inference times are reported alongside the training-speed comparison.
  • Failure cases: Incorrect examples involve severe occlusion, confusion between similar-looking characters, and text orientations curved nearly ninety degrees from horizontal.These failure types are identified as future research directions.
Loading 1806.00926v2…