PaperlayerView exact revision on arXiv

Source-linked AI summary

On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention

Junyeop Lee, Sungrae Park, Jeonghun Baek, Seong Joon Oh, Seonghyeon Kim, Hwalsuk Lee

arXiv:1910.04396v1cs.CV

TL;DR

Scene text recognition still struggles with heavily curved, rotated, and otherwise arbitrary-shaped text because existing methods assume canonical layouts or constrain spatial transformations. SATRN uses Transformer-inspired 2D self-attention with modified encoder modules to model long-range spatial dependencies, and it delivers a 5.7 pp average gain over prior state of the art on irregular benchmarks.

  • Problem

    Existing scene text recognition methods struggle with arbitrary-shaped text, while conventional architectures assume horizontal layouts or require predefined transformations.

  • Method

    SATRN applies self-attention to two-dimensional feature maps and modifies the Transformer encoder with a shallow CNN, adaptive 2D positional encoding, and a locality-aware feedforward layer.

  • Results

    SATRN is state of the art on five of seven benchmarks and improves average performance by 5.7 pp over prior state of the art on irregular benchmarks.

  • Takeaways & Limitations

    SATRN particularly supports recognition of texts with extreme or noncanonical shapes, including rotated and multi-line cases evaluated by the paper.

  • Takeaways & Limitations

    Input rectification methods remain limited when their possible transformation family must be specified beforehand.

Abstract

from arXiv · show

Scene text recognition (STR) is the task of recognizing character sequences in natural scenes. While there have been great advances in STR methods, current methods still fail to recognize texts in arbitrary shapes, such as heavily curved or rotated texts, which are abundant in daily life (e.g. restaurant signs, product labels, company logos, etc). This paper introduces a novel architecture to recognizing texts of arbitrary shapes, named Self-Attention Text Recognition Network (SATRN), which is inspired by the Transformer. SATRN utilizes the self-attention mechanism to describe two-dimensional (2D) spatial dependencies of characters in a scene text image. Exploiting the full-graph propagation of self-attention, SATRN can recognize texts with arbitrary arrangements and large inter-character spacing. As a result, SATRN outperforms existing STR models by a large margin of 5.7 pp on average in "irregular text" benchmarks. We provide empirical analyses that illustrate the inner mechanisms and the extent to which the model is applicable (e.g. rotated and multi-line text). We will open-source the code.

Introduction

Scene text recognition remains limited on arbitrary-shaped text because conventional models assume horizontal layouts or collapse 2D features. SATRN addresses this gap with 2D self-attention and Transformer-based encoder modifications, achieving strong irregular-text results.

  • Problem: Conventional STR models assume horizontal text and are poorly suited to arbitrary shapes encountered in realistic deployment.Many methods collapse the height dimension of 2D CNN features into a 1D sequence.
  • Approach: SATRN preserves two-dimensional feature maps and uses self-attention to model long-range spatial dependencies across irregular text layouts.This design supports character arrangements that do not follow canonical horizontal order.
  • Prior limitations: Existing alternatives either require predefined transformation families, retain horizontal assumptions, use overly complicated structures, or require character bounding boxes.These limitations motivate a simpler architecture that directly handles arbitrary shapes.
  • Approach: The architecture modifies the Transformer encoder with a shallow CNN, adaptive 2D positional encoding, and a locality-aware feedforward layer.These modules are intended to make self-attention effective and efficient on 2D feature maps.
  • Results: SATRN achieves state-of-the-art performance on five of seven benchmarks, with a 5.7 pp average boost over prior state of the art on irregular benchmarks.The model is also evaluated on rotated and multi-line text and computational cost.
  • Contribution: SATRN combines a Transformer-inspired recognition architecture with experimental analysis of its modules and performance on extreme text shapes.The paper presents SATRN as architecturally simple, memory efficient, and accurate.

Related Works

Prior STR methods address noncanonical text through rectification or 2D feature processing, but each retains important constraints. SATRN instead applies self-attention to preserve and connect spatial information across two-dimensional feature maps.

  • Prior STR methods: Earlier STR systems extract width-directional 1D features and therefore fail on curved or rotated text by design.Spatial transformer networks improve noncanonical handling but remain constrained by their transformation space and interpolation losses.
  • 2D feature methods: 2D-feature approaches retain spatial maps but impose constraints such as predefined directions, expensive character-level supervision, or horizontal bias from height pooling and recurrent decoding.These methods sequentially attend to selected regions while following character order.
  • Transformer background: Transformers model long-range pairwise dependencies through self-attention without recurrent modules, motivating their adaptation to visual tasks.Vision adaptations reduce pairwise cost using convolutional layers or pair pruning.
  • SATRN: SATRN adopts self-attention for STR while keeping intermediate feature maps two dimensional, enabling spatial dependency modeling across irregular geometries.Its design applies Transformer techniques to scene text recognition rather than treating the image as a purely one-dimensional sequence.

SATRN Method

SATRN adapts the Transformer for arbitrary-shaped scene text by preserving 2D visual structure and enriching it with spatially adaptive self-attention. Its encoder combines shallow convolution, adaptive 2D positional encoding, and locality-aware feedforward layers before a decoder generates characters.

  • Architecture: SATRN encodes an image into a 2D feature map and decodes that representation into a character sequence.The encoder occupies the architecture’s left side, while the decoder retrieves enriched visual features to generate text.
  • Encoder design: Three encoder modifications adapt the Transformer to scene text: shallow CNN processing, adaptive 2D positional encoding, and locality-aware feedforward layers.These modules are introduced as necessary modifications for realizing self-attention over 2D feature maps.
  • Shallow CNN: The shallow CNN extracts low-level patterns and reduces spatial dimensions before expensive self-attention processing.Two convolution-and-pooling stages provide a 1/4 reduction factor intended to balance computation and performance.
  • Adaptive 2D positional encoding: Adaptive 2D positional encoding dynamically adjusts height-versus-width positional scales to accommodate diverse character arrangements.The scales are inferred from the input feature map, allowing positional encoding to adapt to the spatial diversity of text.
  • Self-attention and locality: Self-attention models dependencies across the 2D feature map, while locality-aware feedforward layers add local structural focus through 3 × 3 convolutions.The feedforward replacement targets the gap between self-attention’s long-range modeling and local character structure.

Experiments

The experiments evaluate SATRN through benchmark accuracy, spatial-dependency analysis, computational efficiency, ablations, and additional assessments of applicability.

  • Experimental program: SATRN is evaluated against state-of-the-art accuracy, spatial dependencies, computational efficiency, design ablations, and broader applicability.The planned analyses cover memory consumption, FLOPs, and the shallow CNN, adaptive positional encoding, and locality-aware feedforward layer.

STR Benchmark Datasets

The evaluation uses seven real-world STR benchmarks divided into regular horizontal-text and irregular arbitrary-shape groups. The benchmark suite includes perspective and heavily curved text cases.

  • Benchmark organization: Seven real-world STR benchmarks are divided into regular horizontally aligned datasets and irregular datasets with more diverse text shapes.The suite follows the regular-versus-irregular grouping used for evaluating scene text recognition difficulty and geometry.
  • Irregular benchmarks: The irregular benchmarks include ICDAR2015, perspective-view SVTP, and heavily curved CUTE80 text images.ICDAR2015 contains more irregular examples than earlier datasets, while SVTP and CUTE80 target perspective and curved text respectively.

Implementation Details

SATRN is trained on combined synthetic text data with standardized image sizing, end-to-end optimization, rotation augmentation, and case-insensitive word-accuracy evaluation.

  • Training data: SATRN is trained on the combined SynthText+Mjsynth dataset for fair comparison with recent STR work.Mjsynth contains 9 million synthetic samples, while SynthText contains 8 million text boxes from 800,000 synthetic scene images.
  • Model configuration: Input images are resized to 32 × 100, with 12 encoder and 6 decoder self-attention layers and a 94-class output vocabulary.The output includes digits, alphabets, special characters, and an end token.
  • Optimization: Training uses end-to-end cross-entropy optimization with rotation augmentation sampled from N(0, (34°)^2) and Adam optimization.The learning rate starts at 3e-4, uses cyclic scheduling, batch size 256, and ends after 4 epochs.
  • Evaluation protocol: Evaluation reports case-insensitive word accuracy after training with special characters, while ablations use a unified 8,539-image benchmark set.The evaluation protocol follows recent STR studies and applies the same optimization method to baseline models in ablations.

Comparison against Prior STR Methods

SATRN is compared with prior STR methods using benchmark accuracy and efficiency evidence, including Table 2’s encoder-decoder analysis and Figure 5’s accuracy-cost comparison.

  • Comparison against Prior STR Methods: SATRN attains the best performance on five of seven benchmarks and improves over the second-best method by 4.7 pp on average on irregular benchmarks.The comparison is reported in Table 1 against prior STR methods.
  • Comparison against Prior STR Methods: Table 2 compares SAR and SATRN by accuracy, parameter count, and FLOPs, with SAR as the first row and SATRN as the final proposed model.
  • Comparison against Prior STR Methods: Figure 5 presents accuracy-efficiency trade-off plots for small, middle, and big SAR and SATRN variants controlled by layer count.

Comparing SATRN against SAR

The SAR comparison isolates the effects of SATRN’s self-attention encoder and decoder, showing improved accuracy-efficiency trade-offs and long-range feature propagation.

  • Comparing SATRN against SAR: Replacing ResNet with the SATRN encoder improves accuracy by 1.0 pp and 0.9 pp while reducing parameters by 12M and FLOPs by 5.5B.The gains are reported for comparisons involving LSTM and SATRN decoders.
  • Comparing SATRN against SAR: The SATRN decoder adds a further 0.3 pp accuracy gain but increases memory consumption by 11M and FLOPs by 19.5B.
  • Comparing SATRN against SAR: SATRN provides a better accuracy-efficiency trade-off than SAR across variants with different numbers of layers.Figure 5 is used for the accuracy-cost comparison.
  • Comparing SATRN against SAR: Self-attention propagates supporting signals globally within a small number of layers, including dependencies between adjacent characters.For character ‘M’, the last layer identifies the next character ‘A’, while depth 2 already propagates the signal globally.

Ablation Studies on Proposed Modules

Ablations evaluate SATRN’s adaptive positional encoding, locality-aware feedforward layer, and preservation of 2D feature maps across accuracy and computational costs.

  • Locality-aware feedforward layer: Figure 8 compares feedforward alternatives across parameter counts, FLOPs, and encoder-layer counts.
  • Adaptive 2D positional encoding (A2DPE): A2DPE provides the best accuracy among four positional-encoding options and dynamically adapts grid spacing to horizontal, diagonal, and vertical text.The adaptive spacing reduces representation burden for other modules and is associated with a performance boost.
  • Adaptive 2D positional encoding (A2DPE): The aspect-ratio analysis groups examples using the ratio ||α||1/||β||1, with low ratios mostly horizontal and high ratios mostly vertical.
  • Locality-aware feedforward layer: Depth-wise locality-aware feedforward layers achieve better accuracy at nearly identical computational costs than point-wise feedforward layers.Naive convolution improves accuracy but uses roughly four times more parameters and FLOPs.
  • 2D feature-map resolution: When the feature-map height is downsampled beyond 1/8, performance drops dramatically by more than 2.9 pp.The results emphasize maintaining 2D feature maps throughout computation.

More Challenges: Rotated and Multi-Line Text

SATRN is evaluated on heavily rotated and multi-line text, where its 2D attention supports recognition across orientations and line changes.

  • Rotated text: SATRN outperforms FAN and SAR while retaining stable performance across all tested rotation levels.The models are trained with rotations sampled uniformly from 0° to 360°.
  • Evaluation setup: The evaluation uses synthetic rotated-text training and manually cropped multi-line scene text for evaluation.
  • Multi-line text: SATRN performs better than the baselines on multi-line text, demonstrating the ability to make long-range jumps when changing lines during inference.
  • Multi-line text: The decoder’s 2D attention distinguishes two lines and transitions from the current region to a non-adjacent image region.Figure 9 shows attention following the first line before moving to the next.

Conclusions

SATRN addresses arbitrary-shape scene text recognition by using self-attention to locate characters despite noncanonical arrangements. It achieves state-of-the-art performance on irregular benchmarks and performs well on rotated and multi-line texts.

  • SATRN enables sequential character localization even when text does not follow canonical arrangements by allowing long-range dependencies through self-attention.The architecture adapts the Transformer to scene text recognition with several novel modifications.
  • SATRN achieves a 5.7 pp average boost on irregular text recognition benchmarks over prior state-of-the-art performance.
  • SATRN shows particularly good performance on rotated and multi-line texts, which the paper identifies as future challenges for scene text recognition.
Loading 1910.04396v1…