PaperlayerView exact revision on arXiv

Source-linked AI summary

DPText-DETR: Towards Better Scene Text Detection with Dynamic Points in Transformer

Maoyuan Ye, Jing Zhang, Shanshan Zhao, Juhua Liu, Bo Du, Dacheng Tao

arXiv:2207.04491v2cs.CV

TL;DR

Arbitrary-shaped scene text detection faces coarse positional queries and point labels tied to human reading order, which raises robustness concerns for inverse-like text. DPText-DETR uses explicit, progressively updated point queries, circular shape guidance, and a revised label form; it reports state-of-the-art results on representative benchmarks alongside fast convergence and promising data efficiency.

  • Problem

    Scene text detection remains challenging for arbitrary shapes, while existing Transformer-based methods use coarse positional queries and control-point labels tied to human reading order that can harm robustness on inverse-like text.

    Source: §1 · pp. 1–2

  • Method

    DPText-DETR combines explicit, dynamically updated point queries and EFSA circular shape guidance with a positional label form whose start points are independent of text semantics.

    Source: §1 · p. 2, §3.4 · p. 4

  • Results

    DPText-DETR sets new state-of-the-art on representative arbitrarily-shaped scene text detection benchmarks, with fast convergence and promising data efficiency.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors establish Inverse-Text, a test set intended to facilitate research on detection robustness where inverse-like texts are scarce in existing test sets.

    Source: §1 · p. 2

    The modified spotter suffers from unsynchronized convergence between detection and recognition.

    Source: §4.6 · p. 7

Abstract

from arXiv · show

Recently, Transformer-based methods, which predict polygon points or Bezier curve control points for localizing texts, are popular in scene text detection. However, these methods built upon detection transformer framework might achieve sub-optimal training efficiency and performance due to coarse positional query modeling.In addition, the point label form exploited in previous works implies the reading order of humans, which impedes the detection robustness from our observation. To address these challenges, this paper proposes a concise Dynamic Point Text DEtection TRansformer network, termed DPText-DETR. In detail, DPText-DETR directly leverages explicit point coordinates to generate position queries and dynamically updates them in a progressive way. Moreover, to improve the spatial inductive bias of non-local self-attention in Transformer, we present an Enhanced Factorized Self-Attention module which provides point queries within each instance with circular shape guidance. Furthermore, we design a simple yet effective positional label form to tackle the side effect of the previous form. To further evaluate the impact of different label forms on the detection robustness in real-world scenario, we establish an Inverse-Text test set containing 500 manually labeled images. Extensive experiments prove the high training efficiency, robustness, and state-of-the-art performance of our method on popular benchmarks. The code and the Inverse-Text test set are available at https://github.com/ymy-k/DPText-DETR.

1 Introduction

Arbitrarily shaped scene text remains challenging, and point-based Transformer detectors face coarse positional queries and label forms tied to human reading order. DPText-DETR addresses these issues with explicit, dynamically updated point queries, circular shape guidance, and a revised label form.

  • Scene text detection remains challenging because texts have complex styles and arbitrary shapes.Source: §1 · p. 1
  • Previous control-point labels follow human reading order, a form that harms robustness on inverse-like text despite such text being scarce in existing benchmarks.The paper reports inverse-like text ratios of about 2.8% in Total-Text, 5.2% in CTW1500, and 5.3% in ICDAR2019 ArT.Source: §1 · p. 2
  • EPQM uses explicit point coordinates for positional queries and dynamically updates them, while EFSA adds circular shape guidance to self-attention.Source: §1 · p. 2
  • The authors introduce a positional label form with start points independent of text semantics and establish Inverse-Text to address the scarcity of inverse-like examples in existing datasets.Source: §1 · p. 2
  • DPText-DETR sets new state-of-the-art on representative arbitrarily-shaped scene text detection benchmarks, with fast convergence and promising data efficiency.Source: §1 · p. 2

2 Related Work

Related work develops both Transformer-based detection and contour representations for arbitrary-shaped text. DPText-DETR builds on these lines with explicit point queries and progressive polygon refinement.

2.1 Detection Transformers

  • Detection Transformers evolved to improve convergence and feature use, but box-prediction methods fall short on arbitrary-shaped scene text.Source: §2.1 · p. 2

2.2 Contour-based Text Detection and Spotting

  • Contour methods include Bezier and Fourier fitting, iterative boundary refinement, and Transformer-based point prediction; DPText-DETR instead investigates explicit point queries.Source: §2.2 · p. 2
  • Figure 2 shows DPText-DETR sampling explicit points as positional queries and progressively refining them into polygon predictions.Source: Figure 2 · p. 3

3 Methodology

DPText-DETR combines explicit point queries, progressive point updates, and spatially guided self-attention to model scene-text shapes. It also uses a positional label form based on spatial order rather than text content.

3 Methodology

  • The methodology introduces EPQM and EFSA to build a Transformer-based scene-text detector and investigates how control-point labels influence detection robustness.The section describes the overall pipeline before detailing the components.Source: §3 · p. 3

3.1 Overview

  • A CNN backbone and Transformer encoder produce features and anchor boxes; the decoder refines control points and predicts text classes and coordinates.The decoder combines point-coordinate encodings with content queries, applies EFSA and deformable cross-attention, then updates reference points layer by layer.Source: §3.1 · p. 3

3.2 Positional Label Form

  • The positional label form orders points clockwise and independently of text content, using spatial rules to identify the top and bottom sides.The starting point is adjusted according to which side lies higher or whether the original top side lies at the bottom.Source: §3.2 · p. 3

3.3 Explicit Point Query Modeling

  • EPQM samples explicit point coordinates on proposal-box sides as position priors, then updates those points layer by layer for deformable cross-attention.Unlike the box-based baseline, explicit point priors give control-point queries respective positions; the text shape prior assumes its top and bottom sides can locate it.Source: Prior Points Sampling · p. 4, Point Update · p. 4, Discussion · p. 4

3.4 Enhanced Factorized Self-Attention

  • EFSA augments factorized self-attention with local circular convolution to provide within-instance point queries with circular shape guidance.The module combines intra-instance attention and locally enhanced queries before inter-instance attention; one four-neighborhood circular convolution layer is used as the performance–speed trade-off.Source: §3.4 · p. 4

4 Experiments

DPText-DETR is evaluated on standard arbitrary-shape text benchmarks and robustness test sets, with comparisons, ablations, and further spotter experiments. It reports state-of-the-art benchmark performance and examines how positional labels, EPQM, and EFSA affect robustness, convergence, and data efficiency.

4.1 Datasets

  • The evaluation includes Total-Text, CTW1500, ICDAR19 ArT, Rot.Total-Text, and a 500-image Inverse-Text set with about 40% inverse-like instances.Rot.Total-Text applies large rotations to Total-Text test images; ablations also evaluate on Inverse-Text.Source: §4 · p. 4, §4.1 · pp. 4–5

4.2 Implementation Details

  • Experiments use a ResNet-50 backbone, six encoder and decoder layers, and benchmark-specific pre-training and fine-tuning schedules.Ablations omit pre-training; rotation-augmented models are trained for 200k iterations, while other ablation models run for 120k iterations without rotation augmentation.Source: §4.2 · p. 5

4.3 Comparison with State-of-the-art Methods

  • DPText-DETR outperforms TexBPN++ by 0.5%, 3.3%, and 2.3% F-measure on Total-Text, CTW1500, and ICDAR19 ArT, respectively.It also exceeds FSG by 0.9% and 3.6% on Total-Text and CTW1500, and SwinTextSpotter by 1.0% and 0.8% on those benchmarks.Source: §4.3 · pp. 5–6, Table 1 · p. 5

4.4 Ablation Studies

  • The positional label form improves F-measure by 0.68% on Total-Text, 3.90% on Rot.Total-Text, and 3.07% on Inverse-Text.It also works with rotation augmentation and contributes to faster convergence in that setting.Source: §4.4 · p. 6, Table 2 · p. 6, Figure 5 · p. 7
  • With training data at 25%, adding EFSA to EPQM yields extra F-measure gains of 11.95% on Total-Text and 7.83% on Inverse-Text.EFSA improves F-measure by 0.89% in the Tab. 2 comparison, while EPQM improves performance and makes the major contribution to convergence.Source: §4.4 · p. 6, Table 4 · p. 6

4.5 What Makes Faster Training Convergence?

  • The combined model converges about six times faster than the baseline in the initial training stage, and explicit point updates make a major contribution to EPQM's convergence and performance gains.Point updates refine reference points for deformable cross-attention; explicit sparse-point modeling is identified as key to faster convergence.Source: §4.4 · p. 6, §4.5 · p. 7, Figure 5 · p. 7

4.4 Ablation Studies

  • EFSA is more effective for polygon control points; circular convolution is unsuitable for Bezier controls that may not form a circle and may be far apart.The positional label form mainly improves robustness, while EPQM and EFSA boost performance, convergence, and few-shot learning ability.Source: §4.4 · pp. 6–7

4.6 Further Discussion

  • On Inverse-Text, positional labels improve detection F-measures for ABCNet-v2 and TESTR, and replacing TESTR's detection decoder with DPText-DETR's still works well.The modified spotter has unsynchronized detection and recognition convergence; recognition continues to learn semantics from a coarse text-anchor sub-region.Source: §4.6 · p. 7, Table 6 · p. 7

5 Conclusion

DPText-DETR transforms composite queries into an explicit, complete point formulation and investigates how control-point labels affect detection robustness. The paper reports state-of-the-art performance, training efficiency, and robustness, and establishes Inverse-Text to facilitate future research.

5 Conclusion

  • DPText-DETR transforms composite queries into an explicit and complete point formulation.Source: §5 · p. 7
  • The work investigates control-point labels’ influence on detection robustness and identifies a practical positional label form.Source: §5 · p. 7
  • Extensive experiments demonstrate state-of-the-art performance, training efficiency, and robustness for DPText-DETR.Source: §5 · p. 7
  • The paper establishes the Inverse-Text test set to facilitate future research in this area.Source: §5 · p. 7

A.1 Effectiveness of EPQM on Bezier Variant Detector

EPQM yields a reported 10% gain on F1measure for the Bezier variant detector. The appendix also presents convergence curves for Rot.Total-Text.

A.1 Effectiveness of EPQM on Bezier Variant Detector

  • 10% gains on F1measure are yielded with EPQM on the Bezier variant detector.Source: §A.1 · p. 8
  • Table 3 provides quantitative results on EPQM’s effectiveness for the Bezier variant detector.Source: §A.1 · p. 8
  • Figure 7 presents convergence curves on Rot.Total-Text.Source: Figure 7 · p. 8, §A.1 · p. 8

A.2 Ablations on Circular Convolution

The circular-convolution ablation identifies a one-layer, 4-neighbourhood setting as the best reported trade-off between performance and inference speed, and this setting is used for polygon control-point detectors.

A.2 Ablations on Circular Convolution

  • One layer and a 4-neighbourhood achieve the best reported performance–inference-speed trade-off.Source: §A.2 · p. 8
  • The paper adopts this setting for all polygon control-point version detectors in its experiments.Source: §A.2 · p. 8
  • Table 7 ablates circular convolution; “Adj.” denotes the number of adjacency elements.Source: Table 7 · p. 8

A.3 Ablations on EFSA

The EFSA ablation reports worse overall performance for the Bezier control-point detector, with circular convolution conjectured to add noise when its points do not form circular shapes. The Inverse-Text supplement describes sample images and statistics, including distributions similar to Total-Text.

A.3 Ablations on EFSA

  • EFSA impedes overall performance for the Bezier control-point detector; the authors conjecture circular convolution adds noise when its points are not circular or are far apart.Source: §A.3 · pp. 8–9

B More Details about Inverse-Text

B.1 Statistical Details

  • Inverse-Text and Total-Text have similar distributions of cared-instance counts, text lengths, and character classes.Most images contain no more than 15 cared text instances; text-length and character-frequency distributions are roughly in line with the English corpus.Source: §B.1 · p. 9

B.2 More Samples

Figure 9 presents additional samples.

B.3 Results of Some Spotters

The section evaluates several spotters on Inverse-Text and reports that they struggle to recognize inverse-like text because of its unconventional reading order.

  • Table 9 reports Inverse-Text spotting results without a lexicon (“None”) and with a lexicon (“Full”).Source: Table 9 · p. 9
  • Figure 10 presents qualitative Inverse-Text results.Source: Figure 10 · p. 9, §B.3 · p. 9
  • The evaluated spotters struggle to recognize inverse-like texts because of their unconventional reading order.The authors say this leaves room for more advanced and robust end-to-end spotters.Source: §B.3 · p. 9
  • The evaluation uses ABCNet-v2 and TESTR as control-point methods, plus segmentation-based SwinTextSpotter, with released weights trained on Total-Text (Ch’ng, Chan, and Liu 2020).Source: §B.3 · p. 9

Related papers

Loading 2207.04491v2…