PaperlayerView exact revision on arXiv

Source-linked AI summary

TextField: Learning A Deep Direction Field for Irregular Scene Text Detection

Yongchao Xu, Yukang Wang, Wei Zhou, Yongpan Wang, Zhibo Yang, Xiang Bai

arXiv:1812.01393v2cs.CV

TL;DR

Irregular scene text detection remains difficult because simple geometric representations poorly handle curved text, while binary segmentation struggles to separate adjacent instances. TextField learns a per-pixel direction field with a fully convolutional network and uses morphological post-processing for final detection. It substantially improves curved-text detection, remains competitive on multi-oriented datasets, and generalizes to unseen datasets.

  • Problem

    Curved text is poorly handled by rectangle- or quadrilateral-based detectors, while segmentation-based methods struggle to separate adjacent text instances.

  • Method

    TextField learns a two-dimensional direction field pointing away from each text pixel’s nearest boundary, encoding text masks and instance-separation information for morphological post-processing.

  • Results

    80.6% F-measure on Total-Text and 81.4% F-measure on SCUT-CTW1500, with large gains over prior methods on curved text.

  • Takeaways & Limitations

    TextField detects arbitrary-shape text, performs competitively on multi-oriented benchmarks, and generalizes well to unseen datasets.

  • Takeaways & Limitations

    TextField still fails on some images involving occlusion or large character spacing and produces false detections in some text-like areas.

Abstract

from arXiv · show

Scene text detection is an important step of scene text reading system. The main challenges lie on significantly varied sizes and aspect ratios, arbitrary orientations and shapes. Driven by recent progress in deep learning, impressive performances have been achieved for multi-oriented text detection. Yet, the performance drops dramatically in detecting curved texts due to the limited text representation (e.g., horizontal bounding boxes, rotated rectangles, or quadrilaterals). It is of great interest to detect curved texts, which are actually very common in natural scenes. In this paper, we present a novel text detector named TextField for detecting irregular scene texts. Specifically, we learn a direction field pointing away from the nearest text boundary to each text point. This direction field is represented by an image of two-dimensional vectors and learned via a fully convolutional neural network. It encodes both binary text mask and direction information used to separate adjacent text instances, which is challenging for classical segmentation-based approaches. Based on the learned direction field, we apply a simple yet effective morphological-based post-processing to achieve the final detection. Experimental results show that the proposed TextField outperforms the state-of-the-art methods by a large margin (28% and 8%) on two curved text datasets: Total-Text and CTW1500, respectively, and also achieves very competitive performance on multi-oriented datasets: ICDAR 2015 and MSRA-TD500. Furthermore, TextField is robust in generalizing to unseen datasets. The code is available at https://github.com/YukangWang/TextField.

I. INTRODUCTION

TextField addresses the difficulty of detecting irregular and curved scene text, where simple geometric representations fail and binary masks struggle to separate adjacent instances. It learns a direction field that represents arbitrary-shape text and supports instance grouping through post-processing.

  • Scene text detection must handle varied sizes, aspect ratios, lighting, orientations, and shapes while supporting downstream text recognition.
  • Deep regression and hybrid methods perform well on standard benchmarks but rely on rectangles or quadrilaterals that poorly represent irregular text.
  • Segmentation-based methods represent arbitrary text areas but often require heavy post-processing to separate nearby text instances.
  • Around 40% of text instances are curved in Total-Text and SCUT-CTW1500, underscoring the practical relevance of irregular-text detection.
  • TextField learns a two-dimensional direction field whose magnitude encodes the text mask and whose direction supports separating text parts into instances.
  • TextField significantly outperforms state-of-the-art methods on two curved-text datasets and remains competitive on ICDAR 2015 and MSRA-TD500.

II. RELATED WORKS

Prior scene text detectors include engineered pipelines, regression, segmentation, hybrid, and part-based methods, each trading representation flexibility against instance separation or pipeline complexity. TextField is positioned as a direction-field alternative for irregular shapes, with morphological processing in its detection pipeline.

  • Traditional component-based detectors use engineered features, heuristic grouping, parameter tuning, and multi-stage pipelines that can reduce performance and speed.
  • Proposal-based and hybrid methods commonly regress horizontal, oriented, or quadrilateral representations, limiting their ability to detect curved text.
  • Part-based methods regress text segments and linking relationships, offering more flexible representations but often producing multi-oriented detections.
  • Segmentation-based methods predict text regions with fully convolutional networks, then apply substantial post-processing to form words or text lines.
  • TextField uses a learned direction field to encode text masks and grouping information, enabling accurate detection of irregular shapes.
  • Its pipeline predicts the field, visualizes magnitude and direction information, and obtains text instances through morphological tools.

III. PROPOSED METHODOLOGY

TextField treats detection as instance segmentation while augmenting a text mask with per-pixel direction information. A fully convolutional VGG16-based network predicts this field, and morphological post-processing groups pixels into text instances.

  • A. Overview: The field is learned with a fully convolutional VGG16-based network using multi-level feature fusion for multi-scale text instances.
  • A. Overview: The network directly regresses a dense per-pixel direction field, with training adaptations designed to emphasize hard pixels and address foreground-background imbalance.
  • B. Direction field: The direction field encodes both a binary text mask and directional information that helps separate adjacent text instances.
  • B. Direction field: For each text pixel, TextField defines a unit vector pointing away from the nearest text-boundary pixel; non-text pixels receive (0, 0).
  • B. Direction field: Distance transform computes each text pixel’s nearest outside pixel, making traditional text annotations straightforward to convert into direction fields.

C. Network architecture

TextField uses a fully convolutional, multi-level feature network to predict a two-channel direction field for irregular text detection. Its local boundary-based representation supports flexibility for long, irregular texts.

  • C. Network architecture: The network uses VGG16 feature extraction with multi-level fusion from stages 3, 4, and 5 to capture text instances at varied scales.Features are upsampled, concatenated, processed by three convolution layers, and converted into a two-channel direction-field map.
  • C. Network architecture: The network predicts the direction field with a fully convolutional architecture and upsamples it to the original image size.Bilinear interpolation is used for the upsampling operations.
  • C. Network architecture: The direction field relies on the nearest text boundary, so the receptive field only needs to cover the text instance’s short side.This design is described as more flexible for detecting irregular long texts than proposal-based methods requiring larger receptive fields.
  • C. Network architecture: Figure 6 presents post-processing stages from candidate-pixel directions through text superpixels, grouped representatives, filtering, and final segmented instances.The figure illustrates the processing sequence on a test image.

D. Optimization

Optimization trains the network to regress the direction field with instance-balanced pixel loss and hard negative mining, followed by morphological inference that groups and filters text regions.

  • 1) Training objective:: The direction field is learned with an instance-balanced Euclidean loss, implemented as a weighted mean squared error over image pixels.The formulation uses the predicted direction field and per-pixel weights.
  • 1) Training objective:: Instance-balanced weighting gives text instances of different sizes equal contributions to the loss.For a pixel p in instance T_p, the weighting depends on the total size of T and the size of T_p.
  • 2) Online hard negative mining:: Hard negative mining retains high-loss non-text pixels to address the imbalance between the many non-text pixels and relatively few text pixels.Non-text pixels are sorted by per-pixel loss, and only a specified front portion is reserved for backpropagation.
  • E. Inference and post-processing: Inference thresholds direction-field magnitudes for candidate text pixels, uses binned directions to build a parent forest, and segments candidates into text superpixels.Each candidate pixel points to a unique neighboring candidate pixel, enabling efficient forest-based partitioning.
  • E. Inference and post-processing: Morphological post-processing dilates superpixel representatives, labels candidate instances, filters unbalanced representatives, closes holes, and removes small instances.The final labels are propagated through text superpixels before closing and size filtering.

IV. EXPERIMENTS

Experiments evaluate TextField on curved-text, multi-oriented, and synthetic pretraining datasets. The figures visualize the learned direction field and successive post-processing stages across benchmark datasets.

  • IV. EXPERIMENTS: TextField is evaluated on SCUT-CTW1500, Total-Text, ICDAR2015, and MSRA-TD500, with SynthText used to pre-train the network.The benchmark set covers curved and multi-oriented scene text.
  • IV. EXPERIMENTS: Figure 7 compares input images, candidate-pixel directions, text superpixels and representatives, filtered labels, and final segmented instances.Panels show examples from SCUT-CTW500, Total-Text, IC15, and MSRA-TD500.

A. Datasets and evaluation protocol

The experiments use synthetic pretraining and four public benchmarks spanning curved, arbitrary-shaped, and multi-oriented text. Performance follows the standard precision, recall, and F-measure protocol based on instance overlap.

  • A. Datasets and evaluation protocol: SynthText provides 800k synthetic images with character-, word-, and line-level annotations; its word-level annotations pre-train TextField.The images blend natural scenes with artificial text.
  • A. Datasets and evaluation protocol: SCUT-CTW1500 contains 1,000 training and 500 testing images, with more than 10k polygon annotations and at least one curved text per image.Text instances are labeled with 14-point polygons at line or curve level.
  • A. Datasets and evaluation protocol: Total-Text contains 1,555 images with curved and multi-oriented texts annotated using word-level polygon-shaped bounding boxes.The split includes 1,255 training images and 300 testing images.
  • A. Datasets and evaluation protocol: ICDAR2015 benchmarks multi-oriented detection under varied scales, orientations, contrast, blur, and viewpoint from incidental Google Glass imagery.These conditions make detection challenging.
  • A. Datasets and evaluation protocol: MSRA-TD500 targets multilingual, arbitrarily oriented long texts with line-level annotations and uses HUST-TR400 as extra training data.The dataset contains 300 training and 200 testing images.
  • A. Datasets and evaluation protocol: Evaluation uses precision, recall, and F-measure, counting a detection as correct when its intersection-over-union with a ground-truth instance exceeds typically 0.5.F-measure provides a compromise between precision and recall.

B. Implementation Details

Training uses augmentation, SynthText pre-training, and dataset-specific fine-tuning to improve robustness and reduce over-fitting.

  • Images are randomly cropped and rotated during augmentation, with crops retained only when contained text exceeds a random 0.1 area threshold.
  • The network is pre-trained on SynthText for one epoch before being fine-tuned separately on four target datasets.

C. Curved text detection

TextField detects curved text accurately at both line and word levels, achieving state-of-the-art results on SCUT-CTW1500 and Total-Text.

  • 81.4% F-measure on SCUT-CTW1500 improves over state-of-the-art methods by 8.0%.
  • 80.6% F-measure on Total-Text significantly outperforms other methods.
  • TextField accurately detects arbitrary-shaped text with precise boundaries and separates close text instances.
  • The method establishes new state-of-the-art results for curved text detection across line-level and word-level evaluations.

D. Multi-oriented text detection

TextField remains competitive on multi-oriented text benchmarks, detecting challenging text across varying orientations, scales, contrasts, and annotation levels.

  • ICDAR2015 evaluation uses original-resolution images because the dataset contains low-resolution images with many small text instances.
  • TextField achieves competitive results with state-of-the-art methods on ICDAR2015, including under multi-scale evaluation.
  • TextField successfully detects long MSRA-TD500 text lines across arbitrary orientations and sizes.
  • On MSRA-TD500, TextField performs slightly worse than methods in and, while performing much better than them on IC15.
  • TextField accurately detects multi-oriented texts at both line and word levels, demonstrating versatility.

E. Cross dataset text detection

Cross-dataset experiments evaluate transfer between datasets with matching annotation levels and show robust generalization to unseen datasets.

  • TextField generalizes better in cross-dataset text detection when trained on one dataset and tested on another with the same annotation level.
  • A model trained on MSRA-TD500 performs comparably on SCUT-CTW1500 with methods trained directly on the target dataset.
  • Cross-dataset experiments show TextField is effective for irregular texts and robust in generalizing to unseen datasets.

F. Runtime

TextField combines network inference with morphological post-processing, running at 6.0 FPS with VGG16 while detecting irregular text and generalizing to unseen datasets. It remains vulnerable to occlusion, large character spacing, and text-like regions.

  • Runtime: TextField’s runtime comprises network inference followed by morphological post-processing to produce final detections.The post-processing stage takes about 36 ms for a 1280 × 720 ICDAR 2015 image and 24 ms for a 768 × 768 MSRA-TD500 image.
  • Runtime: 6.0 FPS: TextField runs at this speed with a VGG16 backbone, on par with most state-of-the-art methods.Inference takes about 130 ms for a 1280 × 720 ICDAR 2015 image, while post-processing takes about 36 ms on the same resolution.
  • Generalization: TextField accurately detects irregular texts and generalizes well to unseen datasets.Cross-dataset evaluations support its generalization to unseen datasets.
  • Failure cases: TextField still fails on difficult images involving object occlusion and large character spacing.The paper also reports false detections on some text-like areas.
  • Failure cases: Figure 9 marks correct detections with green contours, missing ground truths with red contours, and false detections with blue contours.These color encodings identify the main error types in the reported failure examples.
Loading 1812.01393v2…