PaperlayerView exact revision on arXiv

Source-linked AI summary

Text Spotting Transformers

Xiang Zhang, Yongwen Su, Subarna Tripathi, Zhuowen Tu

arXiv:2204.01918v1cs.CV

TL;DR

Scene-text spotting must detect and read text despite wide variation in natural images, and arbitrary orientations and non-rectangular shapes often require heuristic intermediate steps. TESTR unifies detection and recognition with Transformer decoders and control-point representations; experiments report state-of-the-art performance on Total-Text and CTW1500.

  • Problem

    Scene text spotting must both localize text boxes and read characters despite natural-image variation in font, size, style, shape, occlusion, and layout.

    Source: §1 · p. 1

  • Method

    TESTR unifies text detection and recognition with a single encoder, dual decoders, direct control-point regression, and bounding-box-guided polygon detection.

    Source: §1 · pp. 1–2

  • Results

    TESTR reports state-of-the-art performance on Total-Text and CTW1500; on CTW-1500 detection, its F-score exceeds ABCNet v2 by 1.6% (Bezier) and 2.4% (polygonal annotations).

    Source: §1 · p. 2, §3.3 · p. 6

  • Takeaways & Limitations

    The canonical control-point representation supports both polygonal and Bezier annotations, while box-guided polygon detection allows efficient detection of arbitrarily shaped text.

    Source: §4 · p. 9

    The method assumes a fixed number of polygon control points, which may not suit both mostly perspective text and more highly curved text.

    Source: §4 · p. 9

Abstract

from arXiv · show

In this paper, we present TExt Spotting TRansformers (TESTR), a generic end-to-end text spotting framework using Transformers for text detection and recognition in the wild. TESTR builds upon a single encoder and dual decoders for the joint text-box control point regression and character recognition. Other than most existing literature, our method is free from Region-of-Interest operations and heuristics-driven post-processing procedures; TESTR is particularly effective when dealing with curved text-boxes where special cares are needed for the adaptation of the traditional bounding-box representations. We show our canonical representation of control points suitable for text instances in both Bezier curve and polygon annotations. In addition, we design a bounding-box guided polygon detection (box-to-polygon) process. Experiments on curved and arbitrarily shaped datasets demonstrate state-of-the-art performances of the proposed TESTR algorithm.

1. Introduction

Text spotting must localize and read highly variable scene text, while arbitrary orientations and shapes challenge conventional box-based pipelines. TESTR unifies detection and recognition with Transformer decoders, control-point outputs, and box-guided polygon detection.

1. Introduction

  • Natural-scene text varies in font, size, style, shape, and layout; arbitrary orientations and non-rectangular boxes often call for heuristic intermediate or post-processing steps.Source: §1 · p. 1
  • TESTR jointly detects and recognizes text with a single encoder and dual decoders, directly regressing control points without RoI operations or heuristic post-processing.Source: §1 · pp. 1–2
  • The box-to-polygon process adds bounding-box-guided polygon detection, and the canonical control-point representation supports polygonal and Bezier annotations.Source: §1 · p. 2
  • TESTR achieves state-of-the-art performance on Total-Text and CTW1500.Source: §1 · p. 2

1.1. Related Works

  • Earlier detection work explored rotated boxes, character boxes, pairwise points, and Bezier curves, while recognition methods used spatial transformers to rectify irregular text.Source: §1.1 · p. 2
  • End-to-end approaches for irregular text retained shape limits or extra operations such as RoI transforms, grouping, polygon restoration, and post-processing.Source: §1.1 · pp. 2–3

2. Method

TESTR is an end-to-end framework for unified text detection and recognition. It combines multi-scale deformable attention and dual decoders with box-guided polygon detection and explicit training losses.

2. Method

  • TESTR is an end-to-end trainable framework that handles text detection and recognition in a unified manner.The method introduces multi-scale deformable attention and develops dual decoders and box-to-polygon detection.Source: §2 · p. 3

2.1. Multi-Scale Deformable Attention

  • Multi-scale deformable attention samples LK points instead of H × W points, reducing computational overhead while enabling multi-scale features for prevalent small text instances.The paper identifies small text instances as an obstacle for text spotting.Source: §2.1 · pp. 3–4

2.2. Dual Decoders

  • Location and character decoders predict text-instance control points and character sequences, using paired queries and factorized self-attention to model query structure.Control points can represent polygon vertices or Bezier curves; factorized attention covers subqueries within an instance and across different instances.Source: §2.2 · p. 4

2.3. Box-to-Polygon Detection Process

  • Top-Q coarse boxes are encoded and added to shared control-point query embeddings, providing image-specific location and scale priors for polygon regression.The encoded box is shared across the instance’s N subqueries and also initializes its reference point for deformable cross-attention.Source: §2.3 · pp. 4–5, Figure 4 · p. 5

2.4. Training Losses

  • Hungarian matching finds an injective assignment between the fixed set of predictions and ground-truth instances, with matching cost combining confidence and control-point deviation.The paper states that the same bipartite matching scheme is used to match guidance-generator proposals with ground-truth boxes.Source: §2.4 · p. 5
  • The decoder objective sums focal instance-classification, L-1 control-point-regression, and cross-entropy character-classification losses.A separate encoder-side loss supervises the bounding-box proposals; bipartite matching pairs predictions with ground-truth instances.Source: §2.4 · p. 5

3. Experiments

Experiments evaluate TESTR on curved and perspective text benchmarks and report benchmark comparisons and component ablations. TESTR-Polygon generally performs better than TESTR-Bezier.

3.1. Datasets

  • Experiments use SynthText 150k, ICDAR 2015, Total-Text, and CTW1500, evaluating predicted polygons against ground truth by IoU.The benchmarks include curved text and perspective text; CTW1500 contains English and Chinese, but Chinese text is ignored during training.Source: §3.1 · p. 6

3.2. Implementation Details

  • The model uses a ResNet-50 backbone, six encoder and decoder layers, and pretraining on SynthText 150k, MLT 2017, and Total-Text before dataset-specific finetuning.For CTW1500, the maximum text length is increased to 100 and finetuning runs for 200k iterations; the other listed datasets use 20k.Source: §3.2 · p. 6

3.3. Results

  • TESTR-Polygon surpasses the best-reported end-to-end results by 2.8% on CTW1500 with full lexicons and by 0.4%/0.3% on Total-Text without/with full lexicons.The paper also reports TESTR-Bezier ahead of the previous most accurate detector by 1.0% F-score on Total-Text; TESTR surpasses ABCNet v2 on CTW-1500 by 1.6% with Bezier and 2.4% with polygonal annotations.Source: §3.3 · p. 6, Table 2 · p. 7, Table 1 · p. 7
  • Qualitative examples show TESTR handling straight and curved text, while the Bezier variant misses “ANKYLOSAURUS” where the polygon variant succeeds.The paper says this failure pattern is consistent with its quantitative results.Source: §3.3 · p. 7, Figure 5 · p. 7
  • On ICDAR 2015, TESTR achieves state-of-the-art detection F-score and competitive end-to-end results, including strong performance without a lexicon.In the lexicon-free setting it is on par with Text Perceptron using generic lexicons; with lexicons, it works best with the Strong type.Source: §3.3 · pp. 7–8

3.4. Ablation Studies

  • Box-to-polygon guidance improves detection Precision, Recall, and F-score by 0.5%, 3.6%, and 2.2%, respectively, and end-to-end recognition by 5.8%.The guidance generator produces coarse boxes that are added to control-point query embeddings to guide location-decoder regression.Source: §3.4 · p. 8, Table 4 · p. 8
  • Multi-scale features improve detection Precision, Recall, and F-score by 1.2%, 2.3%, and 1.8%, respectively, and end-to-end results by 10.8%.The authors attribute the benefit to addressing prevalent small text instances and report that recognition benefits from larger-scale features.Source: §3.4 · p. 8, Table 4 · p. 8
  • Larger input scales raise detection and end-to-end F-score while lowering FPS; performance is best with λchar = 8.0 across a tested range of 4.0–10.0.The main experiments use λchar = 4.0 to avoid extensive hyperparameter tuning.Source: §3.4 · pp. 8–9, Table 5 · p. 9, Table 6 · p. 9

4. Discussions

TESTR uses a single-encoder, dual-decoder Transformer for holistic text detection and recognition, with box-guided polygon detection and control-point representations for polygonal and Bezier annotations. The paper identifies fixed polygon control-point counts as a limitation and suggests adaptive selection as future work.

4. Discussions

  • TESTR assumes a fixed number of polygon control points, which may be suboptimal because text shapes vary in curvature.The authors propose investigating adaptive selection of the number of points to better capture text shapes.Source: §4 · p. 9
  • The framework jointly predicts text locations and characters without heuristics-driven post-processing or Region-of-Interest operations.Its box-guided polygon detection and canonical control-point representation support polygonal and Bezier annotations.Source: §4 · p. 9
  • The paper reports state-of-the-art performance on Total-Text and CTW1500, challenging benchmarks for curved or oriented text.Source: §4 · p. 9
Loading 2204.01918v1…