PaperlayerView exact revision on arXiv

Source-linked AI summary

All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting

Hao Wang, Pu Lu, Hui Zhang, Mingkun Yang, Xiang Bai, Yongchao Xu, Mengchao He, Yongpan Wang, Wenyu Liu

arXiv:1911.09550v1cs.CV

TL;DR

Existing spotting methods struggle with irregular text, and a prior segmentation-based approach requires character-level annotations. This paper uses boundary points in an end-to-end spotting network and reports state-of-the-art performance in both detection and recognition on standard benchmarks.

  • Problem

    Many existing spotting methods struggle with irregular text, while a segmentation-based arbitrary-shape approach requires extra character-level annotations and adds computational burden.

    Source: Introduction · p. 1

  • Method

    The method localizes text boundaries with points in an end-to-end trainable network, avoiding character-level annotations and supporting joint boundary detection and text recognition.

    Source: Introduction · pp. 1–2

  • Results

    The method achieves state-of-the-art performance in both scene text detection and end-to-end text recognition on standard benchmarks.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    The boundary-point representation is reported to benefit both text detection and recognition on benchmarks covering oriented and curved text.

    Source: Conclusion · p. 7

    The method struggles with rare art fonts, extremely tiny or long text, and blurred text.

    Source: Visualization · pp. 6–7

Abstract

from arXiv · show

Recently, end-to-end text spotting that aims to detect and recognize text from cluttered images simultaneously has received particularly growing interest in computer vision. Different from the existing approaches that formulate text detection as bounding box extraction or instance segmentation, we localize a set of points on the boundary of each text instance. With the representation of such boundary points, we establish a simple yet effective scheme for end-to-end text spotting, which can read the text of arbitrary shapes. Experiments on three challenging datasets, including ICDAR2015, TotalText and COCO-Text demonstrate that the proposed method consistently surpasses the state-of-the-art in both scene text detection and end-to-end text recognition tasks.

Introduction

The proposed method represents text with boundary points in an end-to-end spotting network, addressing irregular shapes without character-level annotations. Its boundary-point approach is reported to achieve state-of-the-art performance in both detection and text spotting on several benchmarks.

Introduction

  • End-to-end spotting combines text detection and recognition, whose relationship can benefit both tasks through feature sharing and joint optimization.Common pipelines detect horizontal or oriented boxes, crop image patches or CNN features, and pass them to a sequence recognizer.Source: Introduction · p. 1
  • Most existing spotting methods struggle with irregular text: rectangular boxes include background, while segmentation-based spotting requires character-level annotations and adds computational burden.The passage notes that rectangles tightly cover straight text but are limited for irregular shapes such as curves.Source: Introduction · p. 1
  • The method localizes text boundaries with points rather than rectangles, enabling an end-to-end trainable network to spot arbitrary-shaped text without character-level annotations.Boundary points can capture irregular-region features, rectify text into horizontal form for sequence recognition, and be refined through recognition training.Source: Introduction · pp. 1–2
  • The method achieves state-of-the-art performance in both text detection and text spotting on several benchmark datasets.The introduction attributes this performance to the boundary-point representation.Source: Introduction · p. 2
  • The paper presents boundary points as a representation for connecting detection and recognition, and jointly optimizes point detection with recognition for straight and curved text.The authors describe boundary points as more suitable than rectangular boxes or segmentation masks for connecting the modules.Source: Introduction · p. 2

Related Work

Prior spotting methods represent text regions with rectangles or segmentation maps, each with limits for curved text or training requirements. The paper positions boundary points as a flexible alternative for end-to-end arbitrary-shape spotting.

Related Work

  • Box-based spotting ranges from separate detection and recognition to shared region features, but rectangular representations handle horizontal or oriented text rather than curved text.Some approaches use horizontal boxes, while others use minimum oriented boxes.Source: Related Work · p. 2
  • Segmentation-based spotting can represent curved text, but the cited approach requires character-level annotations and ignores contextual information between characters of a word.The passage identifies Lyu et al. (2018a) as the segmentation-based method discussed.Source: Related Work · p. 2
  • Figure 3 depicts regression from an axis-aligned proposal to an oriented rectangle, then from default points on the minimum rectangle to text boundary points.The figure also depicts aligning the boundary points to the original image.Source: Figure 3 · p. 3
  • Earlier boundary-point methods require complex, non-differentiable post-processing or use a recurrent network described as time-consuming for end-to-end spotting.The passage contrasts these approaches with the proposed representation and method.Source: Related Work · p. 3

Methodology

The pipeline detects oriented boxes, regresses boundary points, and rectifies text features before recognition. Its recognition decoder uses attention over encoded features, while training combines detection, boundary-point, and recognition losses.

Methodology

  • The pipeline uses an oriented-box detector, boundary-point detector, and recognition network, with predicted boundary points rectifying feature maps for recognition.Oriented boxes are produced by predicting each proposal’s center, height, width, and orientation; RotatedRoIAlign transforms features into horizontal ones before boundary-point prediction.Source: Methodology · p. 3

Boundary Point Detection Network

  • BPDN regresses offsets from predefined default points to predict 2K boundary points, and ArbitraryRoIAlign uses them with Thin-Plate-Spline transformation to rectify features.K target points are sampled equidistantly along each long side of the text instance.Source: Boundary Point Detection Network · pp. 3–4

Recognition Network

  • The recognition branch encodes rectified features as a sequence, then an attention-based GRU decoder predicts symbols using attended features and the previous symbol.At each step, the decoder also uses its previous internal state.Source: Recognition Network · p. 4

Loss Functions

  • Training combines RPN and oriented-box losses with Smooth-L1 boundary-point regression and a recognition loss.The boundary-point loss is calculated as Smooth-L1; the RPN loss follows Ren et al. 2015.Source: Loss Functions · p. 4

Experiments

Experiments span horizontal, oriented, and curved text benchmarks. The described datasets provide distinct text orientations, annotations, and evaluation settings.

Experiments

  • The evaluation covers ICDAR2013, ICDAR2015, COCO-Text, and TotalText, spanning horizontal, oriented, and curved text benchmarks.Source: Experiments · p. 4

Datasets

  • SynthText contains about 800,000 synthetically generated images with multi-oriented text and word- and character-level boxes plus transcriptions.Source: Datasets · p. 4
  • TotalText has 1,255 training and 300 test images, with word-level polygon and transcription annotations.Source: Datasets · p. 4
  • ICDAR2015 has 1,000 training and 500 test images, with word-level quadrangles and transcriptions.Source: Datasets · p. 4
  • ICDAR2013 has 229 training and 233 test images, with word- and character-level boxes and transcriptions.Source: Datasets · p. 5
  • COCO-Text contains 63,686 images with varied text orientations, and end-to-end recognition is evaluated without a lexicon.Source: Datasets · p. 5

Implementation Details

Training proceeds from SynthText pretraining to real-data finetuning. The implementation also specifies boundary-point sampling and evaluation-table conventions.

Implementation Details

  • The detector and recognizer are trained end-to-end, first on SynthText and then finetuned on real data.Source: Implementation Details · p. 5
  • Finetuning uses crops resized to 640 × 640 and images from ICDAR2013, ICDAR2015, and TotalText; training uses mini-batches of 16.Source: Implementation Details · p. 5
  • Results tables define detection P, R, and F, end-to-end recognition, and None versus Full lexicon settings.Source: Table 2 · p. 5
  • Boundary-point targets are sampled equidistantly from longer text-boundary sides, with K set to 7.Source: Implementation Details · p. 5

Curved Text

On TotalText, the method reports gains over MaskTextSpotter in detection and end-to-end recognition without a lexicon. The boundary-point representation is described as flexible for locating text instances.

Curved Text

  • TotalText images are resized so their longer sides measure 1,100, and evaluation follows the protocol used by Lyu et al. 2018a.Source: Curved Text · p. 5
  • 3.6% and 12.1% improvements over MaskTextSpotter are reported for detection and end-to-end recognition without a lexicon, respectively.Source: Curved Text · p. 5
  • The accompanying tables cover ICDAR2015 and ICDAR2013, and COCO-Text; COCO-Text end-to-end testing uses no lexicon.Source: Table 3 · p. 6, Table 4 · p. 6
  • Boundary points are described as a more flexible and efficient way to locate text instances.Source: Curved Text · p. 6

Oriented Text

On ICDAR2015, the method reports improvements in detection and end-to-end recognition with a general lexicon. The comparison also distinguishes results under matched training-data settings.

Oriented Text

  • 2.6% and 1.7% improvements in detection and end-to-end tasks with a general lexicon are reported under the MaskTextSpotter-matched setting.Source: Oriented Text · p. 6
  • Against previous methods, the reported gains are 0.6% in detection and 1.1% in end-to-end recognition with a general lexicon.Source: Oriented Text · p. 6
  • Other cited methods used 9,000 additional MLT2017 training images, so the comparison also reports results with MLT2017 excluded.Source: Oriented Text · p. 6

Horizontal Text

On horizontal scene text, the method achieves comparable performance on detection and end-to-end recognition. The section also notes input resizing and differences in training requirements among comparison methods.

Horizontal Text

  • Input images are resized so their longer sides measure 1280 pixels, preserving aspect ratio.Source: Horizontal Text · p. 6
  • The method achieves comparable performance on both horizontal-text tasks.The comparison notes that MaskTextSpotter requires character-level annotations, while FOTS and He et al. use 9,000 extra MLT2017 images.Source: Horizontal Text · p. 6
  • MaskTextSpotter requires character-level annotations, while FOTS and He et al. add 9,000 MLT2017 training images.Source: Horizontal Text · p. 6

Generalization Evaluation

The method achieves state-of-the-art performance on both COCO-Text tasks under the reported evaluation settings. Examples include successful recognition of vertical text, while rare art fonts and tiny, long, or blurred text remain challenging.

Generalization Evaluation

  • On COCO-Text, the method achieves state-of-the-art performance on detection and end-to-end recognition.The model is not trained on the COCO-Text training set; detection uses V1.4 annotations and end-to-end evaluation uses V2.0.Source: Generalization Evaluation · p. 6

Visualization

  • Examples show successful localization and recognition of arbitrary-shaped text, including vertical instances.The examples are from Total-Text, ICDAR2015, and ICDAR2013.Source: Visualization · p. 6, Figure 4 · p. 7
  • Rare art fonts, extremely tiny or long text, and blurred text are reported failure cases.The authors attribute difficulty with rare art fonts to few such samples in the training set.Source: Visualization · pp. 6–7, Figure 5 · p. 7

Ablation Study

Using oriented rather than axis-aligned proposals improves detection and end-to-end recognition in the reported ablations. The authors link oriented proposals to easier, more precise boundary-point prediction.

Ablation Study

  • On ICDAR2015, oriented proposals improve detection by 0.8% and end-to-end recognition by 1.0% over axis-aligned proposals.Source: Ablation Study · p. 7
  • On TotalText, oriented proposals improve detection by 1.2% and end-to-end recognition by 1.2% over axis-aligned proposals.Source: Ablation Study · p. 7
  • The authors say oriented rectangular proposals reduce boundary-point prediction difficulty and improve precision.Source: Ablation Study · p. 7

Conclusion

The paper presents an end-to-end trainable network representing arbitrary-shaped text with boundary points. It reports state-of-the-art results on standard detection and end-to-end recognition benchmarks, while identifying efficiency as future work.

Conclusion

  • The method represents arbitrary-shaped text as a set of boundary points in an end-to-end trainable network.Source: Conclusion · p. 7
  • The method achieves state-of-the-art performance in scene text detection and end-to-end recognition on standard benchmarks, including oriented and curved text.Source: Conclusion · p. 7
  • The authors identify improving boundary-point detection efficiency as future work.Source: Conclusion · p. 7
Loading 1911.09550v1…