Source-linked AI summary
MASTER: Multi-Aspect Non-local Network for Scene Text Recognition
Ning Lu, Wenwen Yu, Xianbiao Qi, Yihao Chen, Ping Gong, Rong Xiao, Xiang Bai
TL;DR
Existing attention-based scene text recognizers face attention drift and poor parallelization, motivating a more robust and efficient design. MASTER uses self-attention to model feature-feature and target-target relationships, with memory-cache inference, and achieves state-of-the-art performance regarding efficiency and recognition while showing robustness to spatial distortions.
Problem
Attention-based RNN encoder-decoder recognizers suffer attention drift and low efficiency from poor parallelization.
Method
MASTER uses multi-aspect self-attention to encode feature-feature and target-target relationships inside the encoder and decoder, together with memory-cache based inference.
Results
MASTER achieves state-of-the-art performances regarding both efficiency and recognition, with greater robustness to spatial distortions and highly parallel training.
Takeaways & Limitations
The proposed approach supports efficient and accurate scene text recognition across regular and irregular text while improving robustness to spatial distortions.
Takeaways & Limitations
The cited discussion identifies vulnerability of existing methods to complicated backgrounds, diverse font types, and irregular character arrangements.
Abstract
from arXiv · showhide
Attention-based scene text recognizers have gained huge success, which leverages a more compact intermediate representation to learn 1d- or 2d- attention by a RNN-based encoder-decoder architecture. However, such methods suffer from attention-drift problem because high similarity among encoded features leads to attention confusion under the RNN-based local attention mechanism. Moreover, RNN-based methods have low efficiency due to poor parallelization. To overcome these problems, we propose the MASTER, a self-attention based scene text recognizer that (1) not only encodes the input-output attention but also learns self-attention which encodes feature-feature and target-target relationships inside the encoder and decoder and (2) learns a more powerful and robust intermediate representation to spatial distortion, and (3) owns a great training efficiency because of high training parallelization and a high-speed inference because of an efficient memory-cache mechanism. Extensive experiments on various benchmarks demonstrate the superior performance of our MASTER on both regular and irregular scene text. Pytorch code can be found at https://github.com/wenwenyu/MASTER-pytorch, and Tensorflow code can be found at https://github.com/jiangxiluning/MASTER-TF.
1. Introduction
Scene text recognition must handle both regular and irregular text, while existing attention-based methods face attention drift and limited efficiency. MASTER addresses these issues with multi-aspect self-attention and memory-cache decoding, targeting accurate and efficient recognition across both settings.
- Irregular scene text is challenging because of curved shapes, perspective distortions, and complex backgrounds.
- Attention-based recognizers can suffer attention drifting, particularly when text lines contain repetitive digits.
- Self-attention encodes global context and can alleviate attention drifting in scene text recognition.
- MASTER combines a multi-aspect non-local block with a CNN backbone to model global context through different aspects of spatial 2D attention.
- A memory-cache decoding strategy removes unnecessary computation and caches intermediate results from previous decoding steps to accelerate inference.
- MASTER achieves state-of-the-art performance on regular and irregular scene text benchmarks, including the best case-sensitive performance on the reported evaluation.
2. Related Works
Scene text recognition has progressed from CNN-RNN and attention-based methods toward approaches handling irregular geometry and global dependencies. MASTER combines input-output 2D attention with self-attention in the feature extractor and decoder using a compact, lower-cost non-local block.
- Regular Text Recognition: CNN-RNN recognizers extract 1D feature sequences and use CTC loss with word-level rather than character-level annotation.The reviewed methods combine CNN feature extraction, RNN sequence encoding, and CTC-based training.
- Attention-Based Recognition: Attention-based encoder-decoder methods address input-target misalignment, but attention can drift in complex scenes or low-quality images.Edit probability estimation targets missing or superfluous characters, while prior work identifies attention drift as a weakness of vanilla 2D attention.
- Irregular Text Recognition: Rectification-based approaches handle irregular text through spatial transformation, but geometric constraints may exaggerate background noise unexpectedly.Methods include spatial transformers, iterative fronto-parallel rectification, line-fitting transformations, and more flexible rectified attention networks.
- Global Dependencies: 2D attention avoids requiring explicit complex spatial transformation, yet it ignores global pixel context and latent dependencies between characters.These omissions motivate modeling relationships beyond the input-output attention alignment.
- MASTER: MASTER learns both 2D input-output attention and self-attention within the feature extractor and decoder through a multi-aspect non-local block with lower computational cost and a compact model.The block is designed to encode different types of spatial feature dependencies while retaining computational efficiency.
3. Methodology
MASTER uses a Multi-Aspect Global Context Attention encoder with a Transformer-based decoder to produce spatially robust feature representations for scene text recognition.
- MASTER combines a Multi-Aspect Global Context Attention based encoder with a Transformer based decoder.
- The encoder maps a fixed-size input image to a sequence of predicted characters through an intermediate feature tensor.
- 3.1. Encoder: The proposed GCAttention is introduced in the encoder to learn a multi-aspect global context representation.
- Global Context Block: A standard global context block performs attention pooling, bottleneck transformation, and broadcast element-wise addition for feature fusion.
- Multi-Aspect GCAttention: MAGC replaces a single attention function with multiple attention functions and concatenates the resulting global contexts.
- Encoder Architecture: The encoder backbone uses residual blocks, MAGC modules, convolution, and pooling, including 1 × 2 pooling to preserve horizontal information for narrow text.
3.2. Decoder
The decoder is a three-block Transformer stack using masked self-attention, encoder-decoder attention, and feed-forward networks for autoregressive character prediction.
- The decoder contains N = 3 fundamental blocks, each comprising masked multi-head attention, multi-head attention, and a feed-forward network.
- Scaled Multi-Head Dot-Product Attention: Scaled dot-product attention maps query vectors and key-value pairs to output representations using attention weights.
- Scaled Multi-Head Dot-Product Attention: Multi-head attention applies different linear transformations, concatenates the heads, and performs a final linear transformation.
- Masked Attention: A lower-triangular mask lets the decoder predict all training time steps simultaneously while restricting each step to previous outputs.
- Feed-Forward Network: The point-wise FFN uses two fully connected layers with a ReLU activation to add non-linearity.
- Prediction and Loss: The model predicts class probabilities with a shared linear classification layer and trains with cross-entropy over decoding positions.
3.3. Memory-Cache based Inference Mechanism
MASTER decodes characters sequentially at inference time but accelerates this process by caching intermediate results and reducing repeated computation.
- Training predicts all time steps in parallel, whereas inference predicts characters sequentially until EOS or the maximum sequence length.
- Algorithm 1: Cached keys and values are appended during decoding, while the CNN feature and position embedding form the decoder input feature.
- Memory-cache based decoding caches intermediate results from previous decoding steps to speed up inference.
- The mechanism removes unnecessary computation and uses a 1D query vector at each decoding step instead of a traditional 2D matrix.
4. Experiments
The experiments evaluate MASTER on multiple scene-text benchmarks and compare it with state-of-the-art methods, including an ablation study.
- The study conducts extensive experiments on several benchmarks to verify the effectiveness of MASTER and compare it with state-of-the-art methods.
- The evaluation introduces the training and testing datasets used in the experiments.
- The paper reports implementation details before presenting detailed comparisons with state-of-the-art methods.
- An ablation study is conducted to analyze the method’s components.
4.1. Datasets
The evaluation uses synthetic training datasets and multiple public benchmarks covering regular and irregular scene text, with varied distortions and lexicon settings.
- Test datasets: The model is evaluated on seven standard benchmarks without using a lexicon.
- Irregular datasets: Irregular benchmarks include IC15, SVT-Perspective, CUTE, and COCO-Text, featuring perspective distortion, curved text, blur, low resolution, and varied text shapes.
- Irregular datasets: COCO-Text contains printed, scanned, and handwritten text with varied line shapes, using 42,618, 9,896, and 9,837 images for training, validation, and testing.
4.2. Network Structure and Implementation Details
MASTER combines a ResNet-based encoder with self-attention components and is trained from scratch primarily on synthetic data, using fixed architectural and decoding settings.
- Network structure: MASTER uses embedding and encoder dimensions of 512, eight multi-head attention heads, feed-forward dimension 2048, and three identical layers.
- Training: The model is trained from scratch on SynthText, Synth90K, and SynthAdd without real-data finetuning, except for COCO-Text.
- Training: Training uses PyTorch, four NVIDIA Tesla V100 GPUs, Adam optimization, cross-entropy loss, batch size 128 × 4, and learning rate 4 × 10^-4.
- Inference: At test time, tall images are evaluated in original and two rotated forms, and the output with maximum probability is selected without a lexicon or beam search.
4.3. Comparisons with State-of-the-arts
MASTER outperforms or closely matches state-of-the-art methods across regular and irregular scene-text benchmarks, while improving robustness to distortion and attention drift.
- Benchmark results: MASTER achieves superior performance on both regular and irregular datasets compared with state-of-the-art methods.
- Regular benchmarks: 3.5% and 6.1% improvements over SAR are reported on IIIT-5K and SVT, respectively.
- Irregular benchmarks: MASTER achieves the best performance on the irregular SVTP and IC15 datasets.
- COCO-Text: On COCO-Text, word accuracy under case-sensitive metrics increases by 1.74%, from 47.35% to 49.09%.
- Robustness: MASTER is more robust than SAR on blurry, curved, or heavily distorted text, where SAR exhibits attention-drift errors.
- Mechanism: The architecture learns feature-feature and target-target self-attention relationships in addition to input-output attention.
4.4. Ablation Studies
Ablations examine the Multi-Aspect Context count, decoder depth, speed, and stability, identifying settings that improve irregular-text recognition and inference efficiency.
- Influence of h: Using the MAGC module consistently improves performance over omitting it, with h = 8 producing gains on all datasets.
- Influence of h: The largest MAGC improvements occur on CUTE, IC15, and SVTP, which are difficult and irregular datasets.
- Influence of N: N = 3 gives the best decoder performance, while N = 6 decreases substantially compared with N = 3.
- Default setting: The default configuration uses h = 8 and N = 3.
- Speed: MASTER runs at 9.2 ms per image compared with 16.1 ms for SAR on 3,000 IIIT-5K test images.
- Speed and stability: MASTER trains faster through parallel training and maintains more stable recognition performance after reaching its best result, although SAR converges faster.
5. Conclusions
MASTER combines a multi-aspect global-context encoder with a transformer-based decoder to model input-output and internal self-attention relationships. Experiments report greater robustness to spatial distortions, highly parallel training, and fast inference through memory caching.
- MASTER uses a Multi-Aspect Global Context Attention encoder and a transformer-based decoder.
- The model jointly learns input-output attention and self-attention encoding feature-feature and target-target relationships.
- Experiments demonstrate that MASTER is more robust to spatial distortions.
- MASTER has a highly parallel and efficient training process, while its inference speed is fast because of a memory-cache mechanism.
- Standard-benchmark experiments report state-of-the-art performance in both efficiency and recognition.