Source-linked AI summary
Real-time Scene Text Detection with Differentiable Binarization
Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, Xiang Bai
TL;DR
Segmentation-based scene text detection handles varied text shapes but depends on costly binarization post-processing. The paper integrates Differentiable Binarization into a segmentation network, enabling adaptive thresholds and joint optimization. Across five benchmarks, the method consistently improves accuracy and speed, including competitive real-time performance with ResNet-18.
Problem
Segmentation-based scene text detectors require binarization post-processing to convert probability maps into text regions, creating a costly inference step.
Method
The paper introduces Differentiable Binarization, an approximate differentiable operation integrated into a segmentation network for adaptive-threshold binarization and joint optimization.
Results
The method achieves consistently better performance on five benchmark datasets and runs faster than previous leading methods.
Takeaways & Limitations
DB supports a robust and fast detector, with competitive performance and real-time inference speed even using a lightweight ResNet-18 backbone.
Takeaways & Limitations
The method cannot handle cases where one text instance is exactly centered inside another text instance.
Abstract
from arXiv · showhide
Recently, segmentation-based methods are quite popular in scene text detection, as the segmentation results can more accurately describe scene text of various shapes such as curve text. However, the post-processing of binarization is essential for segmentation-based detection, which converts probability maps produced by a segmentation method into bounding boxes/regions of text. In this paper, we propose a module named Differentiable Binarization (DB), which can perform the binarization process in a segmentation network. Optimized along with a DB module, a segmentation network can adaptively set the thresholds for binarization, which not only simplifies the post-processing but also enhances the performance of text detection. Based on a simple segmentation network, we validate the performance improvements of DB on five benchmark datasets, which consistently achieves state-of-the-art results, in terms of both detection accuracy and speed. In particular, with a light-weight backbone, the performance improvements by DB are significant so that we can look for an ideal tradeoff between detection accuracy and efficiency. Specifically, with a backbone of ResNet-18, our detector achieves an F-measure of 82.8, running at 62 FPS, on the MSRA-TD500 dataset. Code is available at: https://github.com/MhLiao/DB
Introduction
Scene text detection must localize text across varied shapes, while segmentation-based methods incur costly post-processing. The proposed Differentiable Binarization module integrates adaptive binarization into the network, improving accuracy and speed across benchmarks.
- Scene text detection remains challenging because text instances vary in scale and shape, including horizontal, multi-oriented, and curved text.
- Segmentation-based detectors describe varied text shapes at pixel level but often require complex post-processing that increases inference time.Examples include progressive scale expansion and pixel clustering.
- Differentiable Binarization inserts an approximate, differentiable binarization operation into the segmentation network for joint optimization.The network can predict an adaptive threshold at each image location.
- Our method achieves consistently better performance on five benchmark datasets covering horizontal, multi-oriented, and curved text.
- Our method runs faster than previous leading methods because DB provides a robust binarization map that simplifies post-processing.
- DB significantly enhances detection performance with the lightweight ResNet-18 backbone.
- DB can be removed during inference without sacrificing performance, adding no extra testing memory or time cost.
Related Work
Scene text detectors primarily use regression or segmentation strategies. Regression methods simplify post-processing but struggle with irregular shapes, whereas segmentation methods combine pixel-level predictions with post-processing.
- Regression-based methods directly regress text bounding boxes and usually use simple post-processing such as nonmaximum suppression.
- Regression-based methods are often limited in representing accurate bounding boxes for irregular shapes such as curved text.
- Segmentation-based methods combine pixel-level prediction with post-processing to obtain text bounding boxes.
Methodology
The method integrates differentiable binarization into a segmentation network, enabling adaptive threshold prediction and end-to-end training. It generates supervised probability, binary, and threshold maps, then forms text instances from binarized regions and polygon dilation.
- Network architecture: The architecture predicts probability and threshold maps from cascaded feature-pyramid features, then computes an approximate binary map for training.Supervision is applied to the probability map, threshold map, and approximate binary map.
- Differentiable binarization: Differentiable binarization replaces the non-differentiable fixed-threshold operation with an approximate step function using an adaptively learned threshold map.The amplifying factor k is set to 50 empirically, and the operation remains differentiable for joint optimization.
- Differentiable binarization: Adaptive thresholds help distinguish text regions from background and separate closely joined text instances.The threshold map can highlight text-border regions even without direct supervision, motivating border-like supervision.
- Label generation: Text-instance labels are generated by shrinking annotated polygons for positive regions and using the gap between shrunk and dilated polygons as threshold-map borders.The shrink ratio r is set to 0.4 empirically, and threshold labels use distances to the closest polygon segment.
- Optimization: Training minimizes a weighted sum of probability-map, binary-map, and threshold-map losses, with α=1.0 and β=10.Binary cross-entropy with hard-negative mining is used for the probability and binary maps, while the threshold loss uses L1 distances inside the dilated polygon.
- Inference: Inference binarizes the probability map at a constant threshold of 0.2, extracts connected shrunk regions, and dilates them using a polygon offset.The offset is computed from the shrunk polygon’s area and perimeter, with r′ set to 1.5 empirically.
Experiments
Experiments evaluate DB across five scene-text benchmarks, ablations, qualitative visualizations, and comparisons spanning curved, multi-oriented, and multi-language text. The method consistently combines strong accuracy with faster inference, while ResNet-18 offers a lightweight accuracy–speed tradeoff.
- Experimental setup: 800k SynthText images are used for pre-training, followed by fine-tuning on corresponding real-world datasets.Training uses 100k pre-training iterations and 1200 fine-tuning epochs.
- Ablation study: DB improves F-measure by 3.7% on MSRA-TD500 and 4.9% on CTW1500 with ResNet-18, while adding no inference-time speed cost.With ResNet-50, the gains are 3.2% and 4.6% on the same datasets.
- Curved text detection: DB-ResNet-50 achieves state-of-the-art accuracy and speed on Total-Text and CTW1500, outperforming prior state-of-the-art accuracy by 1.1% and 1.2%.Compared with a segmentation-based detector running at 3.9 FPS on Total-Text, DB-ResNet-50 (800) is 8.2 times faster.
- Multi-oriented text detection: DB-ResNet-50 (1152) achieves state-of-the-art accuracy on ICDAR 2015, while DB-ResNet-18 (736) reaches 48 FPS with an F-measure of 82.3.DB-ResNet-50 (736) outperforms the previous fastest method by 7.2% and runs twice faster.
- Multi-language text detection: On MSRA-TD500, DB-ResNet-18 (736) reaches an F-measure of 82.8 at 62 FPS, closely matching 83.0 from the previous state-of-the-art method.Its speed is 6.2 times higher than the previous fastest method and can reach 82 FPS with a 512-pixel input.
- Limitations: The method has difficulty with text-inside-text cases when one text instance lies exactly in the center of another.This limitation is attributed to the behavior of shrunk text regions and is described as common among segmentation-based detectors.
Conclusion
The paper presents a DB-based framework for arbitrary-shape scene text detection. Across five benchmarks, it reports consistent accuracy and speed advantages, including competitive real-time performance with ResNet-18.
- The framework integrates differentiable binarization into a segmentation network for arbitrary-shape scene text detection.
- ResNet-50 consistently outperforms state-of-the-art methods on five standard scene-text benchmarks in speed and accuracy.
- ResNet-18 provides competitive performance across the testing datasets with real-time inference speed.