PaperlayerView exact revision on arXiv

Source-linked AI summary

Scene Text Recognition with Permuted Autoregressive Sequence Models

Darwin Bautista, Rowel Atienza

arXiv:2207.06966v1cs.CVcs.CL

TL;DR

STR must recognize challenging natural-scene text, while AR and external language-model approaches have directional, rectification, and efficiency limitations. PARSeq uses permutation language modeling to unify shared-weight AR, NAR, and iterative-refinement inference, achieving SOTA accuracy and favorable compute-quality trade-offs across synthetic, real, and challenging datasets.

  • Problem

    STR must handle varied and degraded natural-scene text, while AR models are directional and external language models can rectify correct predictions independently of image features.

  • Method

    PARSeq adapts Permutation Language Modeling to learn shared-weight AR models supporting context-free NAR, context-aware AR, and iterative refinement.

  • Results

    PARSeq achieves SOTA across STR benchmarks, character sets, and challenging real-world datasets, with favorable accuracy versus parameter count, FLOPS, and latency.

  • Takeaways & Limitations

    A single attention-based model combines image and language context while supporting multiple decoding schemes and robustness to vertical and rotated text.

  • Takeaways & Limitations

    Training on all T! token factorizations is computationally infeasible, so PARSeq trains on a sampled subset of permutations.

Abstract

from arXiv · show

Context-aware STR methods typically use internal autoregressive (AR) language models (LM). Inherent limitations of AR models motivated two-stage methods which employ an external LM. The conditional independence of the external LM on the input image may cause it to erroneously rectify correct predictions, leading to significant inefficiencies. Our method, PARSeq, learns an ensemble of internal AR LMs with shared weights using Permutation Language Modeling. It unifies context-free non-AR and context-aware AR inference, and iterative refinement using bidirectional context. Using synthetic training data, PARSeq achieves state-of-the-art (SOTA) results in STR benchmarks (91.9% accuracy) and more challenging datasets. It establishes new SOTA results (96.0% accuracy) when trained on real data. PARSeq is optimal on accuracy vs parameter count, FLOPS, and latency because of its simple, unified structure and parallel token processing. Due to its extensive use of attention, it is robust on arbitrarily-oriented text which is common in real-world images. Code, pretrained weights, and data are available at: https://github.com/baudm/parseq.

1 Introduction

Scene Text Recognition must handle diverse and degraded natural-scene text, motivating context-aware methods that use language semantics. PARSeq addresses limitations of autoregressive and external-language-model approaches by unifying context-free, context-aware, and refinement-based inference.

  • Motivation: STR recognizes cropped natural-scene text despite variation in fonts, orientations, shapes, illumination, occlusion, noise, blur, and distortion.These challenges distinguish STR from document OCR and support applications including self-driving cars, augmented reality, retail, education, and assistive devices.
  • Motivation: Language semantics can aid recognition when occlusion makes image features insufficient for accurate inference.Context-aware methods obtain semantic priors from word representations, dictionaries, or learned sequence models.
  • Limitations of Existing Methods: AR models learn dependencies in one direction and generate tokens serially, causing direction bias and restricting inference to monotonic decoding.The constraint can produce spurious suffixes or direction-dependent predictions.
  • Proposed Approach: PARSeq adapts Permutation Language Modeling to learn shared-weight AR models that support monotonic AR, parallel NAR, and iterative refinement.Attention masks specify token dependencies for different factorizations and decoding schemes.
  • Proposed Approach: PARSeq achieves SOTA across synthetic and real training settings, character sets, and challenging datasets while using parameters, FLOPS, and runtime efficiently.The method jointly processes image and context features in a unified model rather than relying on separate language and fusion models.

2 Related Work

Related STR work separates context-free recognition from context-aware semantic modeling. Existing internal language models are limited to monotonic AR decoding, while external language models provide bidirectional context with erroneous-rectification and computational-overhead concerns.

  • Context-Free STR: Context-free STR predicts conditionally independent characters directly from image features, commonly using CTC-based methods or attention-based alternatives.Ensemble methods can produce initial context-less predictions with attention mechanisms.
  • Context-Aware STR: Context-aware STR methods learn semantic priors with RNNs or Transformers, typically using standard AR training for internal language models.These methods are limited to monotonic AR decoding.
  • Context-Aware STR: External-LM ensembles add bidirectional context for refinement, but image-independent language models can erroneously rectify predictions and incur significant overhead.This limits their usefulness as recognition components.
  • Generalized Sequence Models: Generalized sequence-generation frameworks unify autoregressive and refinement-based non-autoregressive schemes, providing the conceptual basis for unified STR models.Prior work includes PLM- and MLM-related adaptations, while PARSeq applies PLM to STR.

3 Permuted Autoregressive Sequence Models

PARSeq uses a Transformer encoder-decoder trained with permutation language modeling, where attention masks define alternative token dependencies. The resulting shared model supports AR, NAR, and iterative-refinement decoding.

  • Model Architecture: PARSeq uses a 12-layer Vision Transformer encoder and a single-layer decoder to reduce computational requirements with negligible performance impact.The encoder processes image patches, and all output tokens are passed to the decoder.
  • Model Architecture: The decoder combines position, context, and image tokens through attention, then maps hidden states to character logits.Position tokens act as queries and encode target output positions; context-position attention optionally uses a mask.
  • Permutation Language Modeling: Permutation Language Modeling trains the model over alternative factorizations of P(y|x) by enforcing each ordering through attention masks rather than permuting labels.Each permutation specifies a distinct autoregressive model, while standard AR training is the single-canonical-permutation case.
  • Permutation Language Modeling: All possible T! permutations are computationally infeasible, so training samples K structured permutations consisting of paired left-to-right and flipped orderings.The authors report that this sampling procedure produces more stable training.
  • Permutation Language Modeling: Training uses the mean cross-entropy loss over K permutation-derived attention masks.The decoder produces logits for each mask, and padding tokens are ignored in loss computation.
  • Decoding Schemes: PARSeq supports monotonic AR decoding, parallel NAR decoding, and iterative refinement by changing the attention mask and context used at inference.AR generates one token per iteration, NAR generates all tokens simultaneously, and refinement uses all position queries with a cloze mask.

4 Results and Analysis

The experiments evaluate PARSeq across benchmark settings, training regimes, character sets, challenging text conditions, and computational cost. PARSeq achieves strong accuracy and cost–quality trade-offs while supporting unified decoding schemes.

  • Experimental setup: PARSeq experiments cover benchmark datasets, synthetic and real training data, multiple evaluation charsets, challenging text conditions, and computational cost.The evaluation uses word accuracy alongside parameter count, FLOPS, and latency.
  • Ablation on training permutations vs test accuracy: PARSeq requires at least K >= 6 sampled training permutations for all decoding schemes to perform satisfactorily.With K = 1, training reduces to standard AR modeling; NAR decoding fails and cloze accuracy is 71.14%.
  • Comparison to state-of-the-art (SOTA): PARSeqA achieves the highest word accuracies for both synthetic and real training data, while PARSeqN consistently ranks second or third.With real data, PARSeqA establishes new SOTA results.
  • Comparison to state-of-the-art (SOTA): PARSeqA consistently achieves the highest accuracy across 36-, 62-, and 94-character evaluation charsets.Synthetic training shows a steep decline from the 36- to 62- and 94-character settings, suggesting insufficient cased-character diversity.
  • Comparison to state-of-the-art (SOTA): PARSeq is the most robust method against occlusion and text-orientation variability on larger, more challenging datasets.These datasets expose larger accuracy gaps between methods than the standard benchmark.
  • Computational cost: PARSeq-S achieves the highest mean word accuracy and is Pareto-optimal across parameter count, FLOPS, and latency.PARSeq-S uses significantly fewer parameters and FLOPS than ABINet and TRBA, while PARSeq-Ti offers a low-resource alternative.

5 Conclusion

PARSeq adapts permutation language modeling for a unified STR model that supports context-free and context-aware decoding plus iterative refinement. Its attention-based design achieves strong results across character sets and real-world conditions while maintaining favorable computational trade-offs.

  • Conclusion: PARSeq uses adapted permutation language modeling to unify context-free and context-aware decoding with iterative refinement.The model jointly conditions on image and text representations.
  • Conclusion: PARSeq is optimal on accuracy versus parameter count, FLOPS, and latency through unified decoding and parallel Transformer computation.The conclusion attributes this trade-off to the model’s simple structure and parallel computations.
  • Conclusion: PARSeq demonstrates robustness on vertical and rotated text common in real-world images.The conclusion links this robustness to its extensive use of attention.
  • Issues with unidirectionality of AR models in STR: Unidirectional AR models can add spurious suffixes or produce direction-dependent decodings when context is read in only one direction.The examples show these effects even when the input image is fairly clear and horizontal.

B Inefficiency of External Language Models in STR

External language models in STR add computational cost while providing unreliable spelling correction, motivating more efficient internal-language-model designs.

  • External LM reliability: 41.9% top-5 word accuracy and 50.44% top-1 word accuracy show the external LM can produce incorrect outputs even with correct input labels.These measurements concern ABINet’s 36-character spelling-correction setting.
  • Attention context: Scaled dot-product attention computes similarity-weighted transformations of value vectors, optionally restricted by an attention mask.Multi-head attention applies this operation across projected representation subspaces and concatenates the resulting heads.
  • PARSeq architecture: PARSeq uses a largely original ViT encoder and a pre-LayerNorm decoder with more attention heads.The encoder has 12 layers, while the decoder has a single layer.

D.3 Architecture Configuration

PARSeq uses compact ViT-style encoder–decoder components and trains shared-weight autoregressive factorizations across paired permutations to provide complementary context.

  • Architecture Configuration: PARSeq-S is the base model, while PARSeq-Ti scales down the configuration for edge-device suitability.Scaling up only marginally improves benchmark word accuracy, motivating exploration of the smaller variant.
  • Permutation Language Modeling: Masked multi-head attention uses position tokens as queries and context embeddings with positional information as keys and values.Attention masks enforce the conditional dependencies required by each decoding order.
  • Permutation Sampling: Flipped permutation pairs give each element complementary conditioning sets, combining prior-like and context-conditioned probabilities.The authors report smoother and less erratic loss when using flipped pairs.
  • End-of-sequence handling: The [E] token is trained only with left-to-right and right-to-left permutations, supporting both longest-context and no-context decoding.Character tokens are never conditioned on [E].

E.4 Considerations for batched training

Batched training samples permutations from the longest sequence, which can make equivalent permutations occur for shorter padded sequences.

  • Considerations for batched training: Permutations for a mini-batch are always sampled based on its longest sequence.Labels of varying lengths can share a mini-batch despite differing valid permutation counts.
  • Considerations for batched training: Oversampling equivalent permutations for shorter sequences helps training despite increasing redundancy.Grouping sequences shorter than four separately increases training time without improving accuracy or hastening convergence.
  • Considerations for batched training: The simpler batched procedure is retained because separating short sequences does not improve accuracy or convergence speed.The alternative also requires splitting mini-batches further, increasing training time.

F.1 Open Images Datasets

TextOCR and OpenVINO derive from complex Open Images scenes and preserve challenging variation in text resolution, orientation, and quality through minimal filtering.

  • Open Images Datasets: TextOCR and OpenVINO are derived from Open Images, whose scenes average 8.4 objects per image and were not collected specifically for STR.Their source images commonly contain complex scenes with several objects.
  • Open Images Datasets: The datasets contain text with varying resolutions, orientations, and quality.Cropped word boxes from Open Images illustrate this variation.
  • Open Images Datasets: TextOCR and OpenVINO significantly overlap in their source scene images.The overlap is summarized by the number of common source scene images.
  • Dataset preprocessing: The remaining datasets use released archives, original competition data, or reconciled annotations according to dataset-specific procedures.The dataset-usage summary records the resulting training and validation assignments.
  • Dataset preprocessing: TextOCR, OpenVINO, and COCO-Text receive minimal preprocessing, filtering only illegible and non-machine-printed text plus surrounding or duplicate whitespace.This contrasts with the usual removal of non-horizontal text and special characters.

G Training Details

PARSeq training uses fixed schedules and controlled hyperparameter tuning, while evaluation measures accuracy, refinement trade-offs, and latency across decoding schemes and output lengths.

  • Training schedule: 169,680 training iterations use batch size 384, equivalent to 20 real-data epochs and just over 4 synthetic-data epochs.The combined real dataset contains 3,257,585 samples; synthetic training uses MJ+ST with 15.89M samples.
  • Hyperparameter selection: Training comparisons keep hyperparameters fixed except learning rate, which is selected through Median Stopping, Bayesian Optimization, and grid search.The final learning rates are reported in Table 16, with effective scaling based on GPU count and batch size.
  • Refinement evaluation: NAR accuracy gains become insignificant after the second refinement iteration, whereas additional AR iterations provide negligible improvement after the first.The adopted settings are one iteration for AR decoding and two for NAR decoding.
  • Refinement evaluation: PARSeq evaluates word accuracy and single-image latency as the number of refinement iterations changes for each decoding scheme.Figure 11 identifies the refinement-iteration count for each accuracy–latency point.
  • Latency evaluation: NAR methods, including PARSeqN, have near-constant latency across output lengths, while AR latency increases linearly with label length.Latency is measured with PyTorch’s benchmark timer on an NVIDIA Tesla A100 GPU, excluding data-loading and CPU effects.

J Experiments on arbitrarily-oriented text

The experiments examine whether attention-based STR models remain robust to arbitrary text orientation and whether training on arbitrarily-oriented data improves recognition.

  • Orientation robustness: Attention-based models generally recognize arbitrarily-oriented text, while CRNN shows dismal orientation robustness.The study evaluates word accuracy on rotated versions of six benchmark datasets and reports percentage decreases relative to 0° accuracy.
  • Experimental design: The orientation study compares models trained on horizontally-oriented and arbitrarily-oriented TextOCR variants using 0° and rotated benchmark accuracy.Rotated accuracy averages performance at 90°, 180°, and 270°; bold row entries indicate significant differences.
  • Results: Training on arbitrarily-oriented text improves accuracy across models, with larger gains for attention-based models than for CTC-based models.For TRBA and PARSeq, mean accuracy increases are statistically significant, whereas gains are minimal for CTC-based models.
  • Interpretation: The results suggest that image rectification contributes minimally to orientation robustness and that attention is the primary contributor.TRBC is slightly more robust than CRNN but performs badly compared with TRBA.
  • Evaluation scope: The benchmark reports combined word accuracy on 7,672 samples and on a 7,248-sample subset to account for differing test-set compositions.The subset contains IC13 with 857 samples and IC15 with 1,811 samples.

L Qualitative Results

Qualitative evaluations show generally strong recognition on clear text and highlight model-specific behavior on blur, occlusion, distortion, unconventional orientations, and fonts.

  • Regular datasets: On regular datasets, most models recognize clear, horizontal, high-resolution text, while motion blur can make characters ambiguous for every model.No model correctly predicts Verbandstoffe because motion blur makes the character o resemble e.
  • IC15: PARSeq correctly recognizes the telephone number in IC15, while no model resolves epiCentre because of case ambiguity in character C.ABINet and CRNN also struggle with vertically-oriented CONCIERGE and rotated UNSEEN.
  • SVTP: In SVTP, PARSeq produces one of the closest predictions for vertically-oriented REDWOOD but fails on HOUSE when character S is occluded.The analysis attributes the failure to a low-confidence visual feature overpowering textual features in the decoder.
  • CUTE80: On CUTE80, recognition is generally accurate because the samples are high-resolution and good quality, with exceptions mainly involving CRNN and ABINet.The exceptions include BALLYS for ABINet and CLUB and SALMON for CRNN.
  • ArT: On ArT, CRNN fails on vertically-oriented text, while PARSeq confuses W with V in TOMORROW’S under aspect-ratio distortion.Only ViTSTR correctly recognizes FONDENTE, and only TRBA and ViTSTR correctly recognize the nearly upside-down TOMORROW’S.
  • COCO-Text: PARSeq uniquely handles several difficult COCO-Text cases, including occluded characters in Chevron and GUNNESS.All models recognize XT-862K and People, but only PARSeq correctly infers the occluded characters in the harder examples.
  • Additional datasets: PARSeq correctly recognizes all displayed samples from Uber-Text and the additional internet set, including cursive Creative.Uber-Text contains many vertical or rotated outdoor sign texts, while the internet samples use unconventional fonts.
Loading 2207.06966v1…