PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformer for Fast and Efficient Scene Text Recognition

Rowel Atienza

arXiv:2105.08582v1cs.CV

TL;DR

Scene text recognition research has emphasized accuracy more than speed and computational efficiency, despite the importance of resource use for machines operating in natural environments. ViTSTR addresses this gap with a simple single-stage vision-transformer architecture trained end-to-end for character-sequence prediction. Across configurations, it reports competitive or higher accuracy than TRBA with faster processing and lower resource requirements in smaller models, while remaining near accuracy–efficiency frontiers overall.

  • Problem

    Scene text recognition has emphasized accuracy while giving little attention to speed and computational requirements, although machines must read varied natural-scene text.

  • Method

    ViTSTR is a simple single-stage scene text recognizer built from a vision-transformer encoder with a prediction head for parallel character-sequence recognition.

  • Results

    ViTSTR configurations achieve competitive or higher accuracy than TRBA while offering faster processing and, for smaller models, substantially fewer parameters and FLOPS.

  • Takeaways & Limitations

    Balancing accuracy, speed and computational requirements places nearly all ViTSTR configurations at or near the performance frontiers.

  • Takeaways & Limitations

    ViTSTR errors include similar-symbol confusion, scripted fonts, glare, vertical or heavily curved text, and partial occlusion.

Abstract

from arXiv · show

Scene text recognition (STR) enables computers to read text in natural scenes such as object labels, road signs and instructions. STR helps machines perform informed decisions such as what object to pick, which direction to go, and what is the next step of action. In the body of work on STR, the focus has always been on recognition accuracy. There is little emphasis placed on speed and computational efficiency which are equally important especially for energy-constrained mobile machines. In this paper we propose ViTSTR, an STR with a simple single stage model architecture built on a compute and parameter efficient vision transformer (ViT). On a comparable strong baseline method such as TRBA with accuracy of 84.3%, our small ViTSTR achieves a competitive accuracy of 82.6% (84.2% with data augmentation) at 2.4x speed up, using only 43.4% of the number of parameters and 42.2% FLOPS. The tiny version of ViTSTR achieves 80.3% accuracy (82.1% with data augmentation), at 2.5x the speed, requiring only 10.9% of the number of parameters and 11.9% FLOPS. With data augmentation, our base ViTSTR outperforms TRBA at 85.2% accuracy (83.7% without augmentation) at 2.3x the speed but requires 73.2% more parameters and 61.5% more FLOPS. In terms of trade-offs, nearly all ViTSTR configurations are at or near the frontiers to maximize accuracy, speed and computational efficiency all at the same time.

1 Introduction

Scene text recognition must handle highly varied natural-scene text while balancing accuracy with speed and computational efficiency. ViTSTR addresses this goal with a simple vision-transformer architecture and reports strong accuracy–resource trade-offs.

  • Motivation: Scene text recognition supports machine decisions by reading labels, signs, instructions and other text in human environments.Examples include pushing a door, identifying sugar, and reading currency for visually impaired users.
  • Results: Nearly all ViTSTR versions are at or near the frontiers across accuracy, parameter count, speed and FLOPS.The figure uses slope to represent accuracy gain as resources or speed increase; steeper slopes indicate better trade-offs.
  • Challenges: Natural-scene text varies in font style, orientation, shape, size, color, texture, illumination and imaging conditions.These inputs differ from the structured documents typically handled by OCR.
  • Research gap: The field has emphasized recognition accuracy while giving little attention to speed and computing requirements.The paper defines efficiency through parameters and FLOPS, which approximate memory and instruction requirements.
  • Proposed approach: ViTSTR uses a simple single-stage vision-transformer architecture to balance accuracy, speed and computational performance.Its encoder-only design predicts character sequences end-to-end.
  • Results: 80.3% accuracy and 9.3 msec/image are achieved by ViTSTR-Tiny, while ViTSTR-Small reaches 82.6% accuracy at 9.5 msec/image.With data augmentation, the reported accuracies are 82.1% for Tiny and 84.2% for Small.

2 Related Work

Prior STR systems use multiple stages for rectification, feature extraction, sequence modelling and prediction, while transformer-based systems still commonly combine a backbone with an encoder-decoder. ViTSTR simplifies this pattern to a single-stage, encoder-based architecture for sequence recognition.

  • STR setting: Natural-scene text recognition is challenging because text appearance varies with curvature, font style, blur, rotation, noise, geometry, illumination, occlusion and resolution.Weather, camera imperfections, motion and lighting add further variation.
  • STR setting: End-to-end scene text reading generally combines text detection and recognition, whereas this work focuses on recognizing 96 Latin characters.Detection locates text regions; recognition reads the character sequence.
  • STR design: Modern STR design patterns include rectification, feature extraction, sequence modelling and prediction stages.These modules respectively normalize text, extract invariant features, model long-term dependencies and produce character sequences.
  • Transformer approaches: Transformer-based STR models use parallel self-attention and prediction, but current designs still require a backbone and transformer encoder-decoder.ViTSTR instead uses a single stage based on a vision-transformer encoder.

3 Vision Transformer for STR

ViTSTR adapts a vision-transformer encoder for scene text recognition by replacing single-class prediction with parallel sequence prediction. Its architecture uses image patches, positional encoding, stacked encoder blocks, and multiple output projections for word prediction.

  • Architecture: ViTSTR changes only the prediction head from ViT, enabling parallel recognition of multiple characters in their sequence order and length.Unlike single object-class recognition, ViTSTR extracts multiple encoder features for text prediction.
  • Input representation: Image patches are flattened, linearly embedded to width D, prepended with a learnable class embedding, and combined with positional encoding before the encoder.The input image has dimensions H × W × C, and the resulting patch sequence has length N.
  • Sequence prediction: ViTSTR extracts a number of encoder feature vectors equal to the dataset’s maximum text length plus two special tokens.[GO] marks the beginning of prediction, while [s] marks the end or a space and is repeated after the text.
  • Transformer encoder: The encoder is a stack of L identical blocks using layer normalization, multi-head self-attention, MLP feature extraction, and residual connections.Self-attention models relationships among feature vectors, while the MLP contains two GELU-activated layers.
  • Prediction head: A sequence of linear projections forms the word prediction from the encoder outputs.The head produces predictions across positions rather than a single object category.

4 Experimental Results and Discussion

The experiments use a unified evaluation framework to compare ViTSTR with strong STR baselines under consistent train and test conditions. Training relies on synthetic datasets because large real-text datasets are unavailable, with MJ and ST contributing equally to the combined training set.

  • Evaluation framework: A unified framework provides consistent train and test conditions for fair comparison among ViTSTR and strong STR baselines.The reproduced baselines include CRNN, R2AM, GCRNN, Rosetta, RARE, STAR-Net, and TRBA.
  • Training data: Synthetic data is commonly used for STR training because large datasets of real text images are unavailable.The two popular synthetic datasets are MJSynth and SynthText.
  • MJSynth: MJSynth contains 8.9M realistically looking word images generated with varied fonts, rendering properties, backgrounds, distortions, and noise.Its construction uses background, foreground, and optional shadow or border layers.
  • SynthText: SynthText contains 5.5M word images created by blending synthetic text onto natural images using scene geometry, texture, and surface normals.The text is naturally blended and distorted on object surfaces before word images are cropped.
  • Training mixture: Each dataset contributes 50% of the total training set, while combining 100% of both datasets deteriorated performance.Figure 6 shows sample images from MJ and ST.

4.2 Test Dataset

The test data consists of publicly available scene-text datasets divided into regular and irregular categories. Regular images are relatively frontal and minimally distorted, whereas irregular images include curved, vertical, perspective, low-resolution, or distorted text.

  • Dataset organization: The test set comprises several small publicly available STR datasets containing text in natural images.These datasets are grouped into regular and irregular categories.
  • Dataset categories: Regular datasets contain frontal, horizontal text with minimal distortion, while irregular datasets contain more challenging appearances.Irregular examples include curved, vertical, perspective, low-resolution, and distorted text.
  • Evaluation protocol: The evaluation uses only the test splits of the regular and irregular datasets.Figure 7 presents samples from these real-image datasets.

Regular Dataset

The regular-dataset evaluation covers IIIT5K, SVT, IC03, and IC13, which provide natural-scene text images with dataset-specific test-set sizes and versions.

  • IIIT5K: IIIT5K contains 3,000 test images, mostly depicting street scenes such as signboards, brand logos, house numbers, and street signs.
  • SVT: SVT contains 647 test images cropped from Google Street View imagery.
  • IC03: IC03 has 1,110 original test images, reduced to 860 after removing words shorter than three characters, with an 867-image version also included.The additional seven images in the 867-image version address missing images in the framework.
  • IC13: IC13 extends IC03 with similar images and is evaluated in two test-set versions containing 857 or 1,015 images.

Irregular Dataset

The irregular-dataset evaluation uses several challenging natural-image text benchmarks, with standardized training and testing configurations for fair comparison across models.

  • Dataset characteristics: IC15 includes blurry, noisy, rotated, and low-resolution text images captured during unconstrained motion.The framework evaluates both 1,811-image and 2,077-image versions; the larger version includes rotated, vertical, perspective-shifted, and curved images.
  • Dataset characteristics: SVTP contains 645 Google Street View test images, mostly depicting business signage.
  • Dataset characteristics: CT focuses on curved text images captured from shirts and product logos.
  • Evaluation protocol: The evaluation reproduces strong baseline models and trains each model at least five times with different random seeds.The best-performing weights on the test datasets are saved to obtain mean evaluation scores.
  • ViTSTR configuration: ViTSTR uses 224 × 224 inputs, pretrained DeiT weights, and end-to-end training without frozen parameters.
  • Evaluation protocol: Performance is reported using accuracy, speed, parameter count, and FLOPS, with speed measured on a 2080Ti GPU.Accuracy follows case-sensitive training and case-insensitive evaluation; vertical text images are not rotated for benchmarking.

4.4 Data Augmentation

STR-specific image augmentation changes appearance while preserving text meaning, improving ViTSTR accuracy, especially on irregular datasets.

  • Augmentation results: +1.8% accuracy improves ViTSTR-Tiny with RandAugment, compared with +1.6% for Small and +1.5% for Base.
  • Irregular datasets: +9.2% accuracy is the largest reported augmentation gain for ViTSTR-Tiny on CT.Other gains include +6.6% for Small and Base on CT.
  • Augmentation design: RandAugment applies transformations including inversion, curving, blur, noise, distortion, rotation, stretching/compressing, perspective, and shrinking.

4.5 Attention

ViTSTR’s attention focuses on individual characters while also incorporating information from neighboring characters during readout.

  • Attention behavior: ViTSTR attention properly focuses on each character while reading a Nestle text image.
  • Attention behavior: The attention map also shows attention paid to neighboring characters.
  • Attention behavior: The authors suggest that neighboring-character attention may provide context during individual symbol prediction.

4.6 Performance Penalty

ViTSTR scales capacity primarily by enlarging transformer representations rather than adding sequential stages, limiting the associated speed penalty.

  • Multi-stage comparisons: 2.2% accuracy gain from RARE to TRBA requires 38.8M more parameters and adds 4 msec/image.
  • Multi-stage comparisons: Changing CTC to Attention from STAR-Net to TRBA increases accuracy by 2.5% but slows computation from 8.8 to 22.8 msec/image.The passage states this slowdown exceeds 10× the penalty from adding BiLSTM or TPS.
  • ViTSTR scaling: Scaling ViTSTR from Tiny to Small gains 2.3% accuracy with 16.1M additional parameters and only a 0.2–0.3 msec/image slowdown.
  • ViTSTR scaling: Scaling from Tiny to Base gains 3.4% accuracy with 80.4M additional parameters while speed barely changes.
  • Architectural implication: ViTSTR avoids the larger penalty of multi-stage models because its parallel transformer operations do not require additional sequential forward-propagation layers.

5 Conclusion

ViTSTR is a simple single-stage architecture designed to balance recognition accuracy, speed, and computational requirements. Data augmentation improves accuracy, especially on irregular datasets, while scaling preserves this balance near the trade-off frontiers.

  • ViTSTR is a simple single-stage architecture emphasizing balance among accuracy, speed, and computational requirements.
  • Data augmentation can significantly increase ViTSTR accuracy, especially on irregular datasets.
  • When scaled up, ViTSTR remains near the frontiers balancing accuracy, speed, and computational requirements.
Loading 2105.08582v1…