PaperlayerView exact revision on arXiv

Source-linked AI summary

PixelLink: Detecting Scene Text via Instance Segmentation

Dan Deng, Haifeng Liu, Xuelong Li, Deng Cai

arXiv:1801.01315v1cs.CV

TL;DR

Scene-text instances are difficult to separate with semantic segmentation alone, while conventional detectors rely on bounding box regression. PixelLink instead links text pixels into instance segments and extracts boxes directly, achieving comparable or better benchmark performance with less data and training.

  • Problem

    Adjacent scene-text instances are difficult or sometimes impossible to separate using semantic segmentation alone, despite regression not being indispensable for locating text.

  • Method

    PixelLink predicts text pixels and links between neighboring pixels, groups positive predictions into connected components, and extracts bounding boxes directly from the segmentation result.

  • Results

    PixelLink achieves comparable or better performance than regression-based state-of-the-art methods on several benchmarks while requiring fewer training iterations and less training data.

  • Takeaways & Limitations

    PixelLink supports scene-text detection through instance segmentation without location regression, using smaller receptive fields and easier training to enable training from scratch with less data and fewer iterations.

  • Takeaways & Limitations

    Speed comparisons are not fully objective because implementation details and running environments differ, and VGG16 was chosen for convenient comparisons.

Abstract

from arXiv · show

Most state-of-the-art scene text detection algorithms are deep learning based methods that depend on bounding box regression and perform at least two kinds of predictions: text/non-text classification and location regression. Regression plays a key role in the acquisition of bounding boxes in these methods, but it is not indispensable because text/non-text prediction can also be considered as a kind of semantic segmentation that contains full location information in itself. However, text instances in scene images often lie very close to each other, making them very difficult to separate via semantic segmentation. Therefore, instance segmentation is needed to address this problem. In this paper, PixelLink, a novel scene text detection algorithm based on instance segmentation, is proposed. Text instances are first segmented out by linking pixels within the same instance together. Text bounding boxes are then extracted directly from the segmentation result without location regression. Experiments show that, compared with regression-based methods, PixelLink can achieve better or comparable performance on several benchmarks, while requiring many fewer training iterations and less training data.

1 Introduction

PixelLink addresses the difficulty of separating adjacent scene-text instances by replacing location regression with pixel-link-based instance segmentation. It directly extracts text boxes from connected components and achieves comparable or better benchmark performance with less training.

  • Most scene-text detectors combine text/non-text classification with location regression to obtain bounding boxes.
  • Regression is not indispensable because text/non-text score maps contain location information and can directly yield bounding boxes.
  • Adjacent text instances can be difficult or impossible to separate using semantic segmentation alone, motivating instance-level segmentation.
  • PixelLink predicts text pixels and links between neighboring pixels, then groups positive pixels into connected components representing text instances.
  • PixelLink achieves comparable or better performance on several benchmarks from scratch while requiring fewer training iterations and less training data.

2 Related Work

Related work frames scene-text detection through semantic segmentation and regression-based object-detection techniques. Existing methods predict text regions, characters, slices, segments, or boxes and often assemble them with linking or heuristic post-processing.

  • Instance segmentation differs from semantic segmentation by distinguishing individual instances in addition to assigning object-category labels.
  • Segmentation-based text detectors predict maps for text, character classes, or character links, then group candidates into words or lines.
  • Regression-based methods adapt anchors, slices, rotated regions, or text segments from general object detection to scene-text localization.
  • The paper identifies TextBoxes, CTPN, SegLink, and EAST as published state-of-the-art methods on IC13 or IC15.

3 Detecting Text via Instance Segmentation

PixelLink performs scene-text detection by predicting text pixels and eight-direction links, grouping positive predictions into connected components, and extracting oriented boxes directly from them. Post-filtering removes noise without using location regression.

  • 3.1 Network Architecture: PixelLink uses separate pixel-wise heads for text/non-text and link prediction within a CNN architecture.The link head predicts eight directional links, while the text head predicts pixel labels.
  • 3.1 Network Architecture: Two feature-fusion settings, PixelLink+VGG16 2s and PixelLink+VGG16 4s, produce predictions at half and quarter input resolution, respectively.
  • 3.2 Instance Segmentation: Thresholded positive pixels are grouped through positive links into connected components, each representing a detected text instance.Neighboring pixels are connected when one or both corresponding link predictions are positive.
  • 3.3 Extraction of Bounding Boxes: Bounding boxes are extracted from connected components with minAreaRect, producing oriented rectangles convertible to quadrangles or rectangles.The method places no restriction on scene-text orientation.
  • 3.3 Extraction of Bounding Boxes: Bounding boxes come directly from instance segmentation rather than location regression, marking PixelLink’s key design difference.
  • 3.4 Post-filtering: Geometry-based post-filtering removes noisy detections, using criteria such as shorter side, area, width, height, and aspect ratio.In IC15, examples include discarding boxes with shorter side below 10 pixels or area below 300.

4 Optimization

PixelLink optimizes separate pixel and link predictions, balancing instance contributions and hard negatives while combining the two losses. Its training also uses resized ground truth and augmented inputs.

  • Labeling: Pixel labels mark text inside bounding boxes as positive, while links to eight neighbors are positive only when both pixels belong to the same instance.Overlapping text boxes contribute only their un-overlapped pixels as positive; ground truth is computed at the prediction-layer resolution.
  • Loss formulation: The training loss is a weighted sum of pixel and link losses, with λ set to 2.0 because link loss is computed only on positive pixels.This weighting makes pixel classification more important than link classification in the total objective.
  • Pixel loss: Instance-Balanced Cross-Entropy Loss gives every text instance equal total weight, assigning higher per-pixel weights to small instances and lower weights to large ones.This addresses the unfair effect of assigning equal weight to every positive pixel when instance areas vary substantially.
  • Pixel loss: Online Hard Example Mining selects r*S negative pixels with the highest losses, using a negative-positive ratio r of 3.The selected negatives receive unit weights in the pixel-classification weight matrix.
  • Link loss: Link losses for positive and negative links are computed separately on positive pixels using weights derived from the pixel weight matrix.The link objective is a class-balanced cross-entropy loss summed over weighted positive and negative links.
  • Data augmentation: Training augments images with random rotations, crops, and uniform resizing to 512 × 512, while ignoring sufficiently small text instances.Rotation occurs with probability 0.2, and instances with a shorter side below 10 pixels after augmentation are ignored.

5 Experiments

PixelLink is evaluated on several scene-text benchmarks covering arbitrary orientations, horizontal text, and text lines, using standard evaluation protocols. Detection results indicate performance on par with or better than state-of-the-art methods.

  • PixelLink models are evaluated on IC15, IC13, and MSRA-TD500 using their corresponding standard evaluation protocols.IC15 contains arbitrarily oriented text, IC13 mostly horizontal text, and MSRA-TD500 contains longer text lines.
  • PixelLink achieves on-par or better results than state-of-the-art methods across the evaluated benchmarks.The paper presents example detections from IC15, IC13, and MSRA-TD500.
  • IC15 contains 1,000 training images and 500 test images with annotated word quadrilaterals for readable Latin scripts.The benchmark targets scene text in arbitrary directions and ignores annotated “do not care” scripts during evaluation.
  • IC13 contains 229 training images and 233 test images, with mostly horizontal text instances annotated as word rectangles.
  • MSRA-TD500 contains 500 images, including 300 for training and 200 for testing, with arbitrarily oriented English and Chinese text lines.

5.2 Implementation Details

The experiments train PixelLink from randomly initialized VGG models and compare its benchmark results with published methods. The paper reports strong performance but cautions that speed comparisons are not fully objective because implementations and environments differ.

  • Implementation Details: PixelLink is trained with SGD using momentum 0.9 and weight decay 5 × 10^-4, starting from a randomly initialized VGG model rather than ImageNet pretraining.The learning rate is 10^-3 for the first 100 iterations and 10^-2 thereafter.
  • Implementation Details: Training on IC15-train requires about 40K iterations for the 4s model and about 60K iterations for the 2s model.
  • Implementation Details: Table 1 reports Recall, Precision, and F-score for methods tested at 720P or approximately 1280 × 768, with MS denoting multi-scale testing.Comparison results are quoted from the corresponding original papers.
  • Implementation Details: PixelLink’s best IC15 result exceeds EAST+PVA2x, the best existing single-scale method, by 5.5% in F-score.
  • Implementation Details: Objective speed comparison is difficult because implementation details and running environments differ across methods.Reported speeds are intended only to show that PixelLink is not slower when the same VGG16 base network is used.

5.4 Detecting Long Text in TD500

On MSRA-TD500, PixelLink is fine-tuned for detecting text lines and compared with other VGG16-based detectors. The results support its ability to detect long text without a deeper network.

  • 5.4 Detecting Long Text in TD500: For TD500, PixelLink is pretrained on IC15-train for about 15K iterations and fine-tuned on TD500-train plus HUST-TR400 for about 25K iterations.Testing uses 768 × 768 images, pixel and link thresholds of 0.8 and 0.7, and post-filtering thresholds of 15 pixels and area 600.
  • 5.4 Detecting Long Text in TD500: Among VGG16-based models, EAST performs worst, while SegLink and PixelLink detect long texts without requiring a deeper network.The comparison attributes this difference to the models’ differing demands on large receptive fields.

5.5 Detecting Horizontal Text in IC13

For horizontal text in IC13, PixelLink uses a specialized multi-scale fusion scheme because it lacks direct confidence outputs for detected bounding boxes. Multi-scale testing improves F-score by about 4 to 5 points.

  • 5.5 Detecting Horizontal Text in IC13: PixelLink fuses multi-scale prediction maps by resizing them to the largest height and width, then averaging them.The remaining processing steps are identical to single-scale testing.
  • 5.5 Detecting Horizontal Text in IC13: Multi-scale testing improves F-score by about 4 to 5 points on IC13.The paper reports a similar observation for TextBoxes.

6 Analysis and Discussion

PixelLink’s analysis attributes its efficiency and performance to simpler pixel-link predictions, while experiments show that links, filtering, image size, and feature-layer choice materially affect results.

  • The Advantages of PixelLink: PixelLink reaches performance on par with SegLink or EAST after about 25K iterations, using fewer iterations and training from scratch rather than ImageNet pretraining.SegLink trained from scratch on IC15 achieves a 67.81 F-score.
  • Difficulty of tasks: PixelLink’s prediction neurons require smaller receptive fields because they classify local pixels and neighboring links instead of regressing complete bounding-box coordinates.The paper characterizes this as an easier learning task that may reduce training time and data requirements.
  • Difficulty of tasks: Training from scratch with limited data suggests that text detection may rely more on low-level texture features and less on high-level semantic features than general object detection.This interpretation is presented as an indication from PixelLink’s training behavior.
  • Ablation analysis: Disabling links causes a huge drop in recall and precision because links convert semantic segmentation into instance segmentation for separating nearby texts.The ablation identifies link prediction as indispensable for PixelLink’s separation of adjacent text instances.
  • Ablation analysis: Instance-Balance is optional for strong performance: without it, PixelLink achieves an 81.2 F-score, while using it produces a slight improvement.The comparison is reported for IC15, with a larger performance gap observed on IC13.
  • Ablation analysis: Removing post-filtering slightly improves recall but significantly reduces precision, while smaller training images reduce both recall and precision.Using an earlier prediction layer improves recall and precision but reduces speed, exposing a performance–speed tradeoff.

7 Conclusion and Future Work

The paper concludes that PixelLink performs scene text detection through instance segmentation and can match or exceed regression-based methods with less training. Future work includes evaluating other deep models and applying PixelLink to additional instance-segmentation tasks.

  • Conclusion: PixelLink links pixels within each text instance, extracts bounding boxes directly from the segmentation result, and avoids location regression.The conclusion presents this instance-segmentation pipeline as the paper’s central contribution.
  • Conclusion: PixelLink achieves on-par or better performance on several benchmarks while training from scratch with less data and fewer iterations than regression-based methods.The paper attributes this efficiency to smaller receptive-field requirements and easier prediction tasks.
  • Future Work: VGG16 is used as the backbone for convenient comparisons, while other deep models are left for future investigation.The paper explicitly identifies better performance and higher speed as motivations for exploring other models.
  • Future Work: Unlike prevalent instance-segmentation methods, PixelLink does not make its segmentation result depend on detection performance, and applications to other instance-segmentation tasks are proposed.These applications are stated as future exploration rather than demonstrated results in the supplied passage.

Acknowlegement

The authors acknowledge funding from China’s National Natural Science Foundation and express thanks to Dr. Yanwu Xu of CVTE Research.

  • Acknowledgement: The work was supported by the National Natural Science Foundation of China under Grant 61379071.The acknowledgment names the specific grant number.
  • Acknowledgement: The authors thank Dr. Yanwu Xu of CVTE Research for his kindness and help.
Loading 1801.01315v1…