Source-linked AI summary
Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion
Minghui Liao, Zhisheng Zou, Zhaoyi Wan, Cong Yao, Xiang Bai
TL;DR
Segmentation-based scene text detectors handle arbitrary shapes but are limited by complex post-processing and insufficient scale robustness. DBNet++ integrates differentiable binarization and adaptive scale fusion into a segmentation network. It consistently achieves state-of-the-art accuracy and speed across five benchmarks, while remaining limited on text-inside-text cases.
Problem
Segmentation-based scene text detectors face complex, time-consuming post-processing and limited scale robustness despite their pixel-level advantages for irregular text.
Method
DBNet++ jointly optimizes a segmentation network with Differentiable Binarization and adaptively fuses multi-scale features using Adaptive Scale Fusion.
Results
DBNet++ consistently achieves state-of-the-art accuracy on five standard benchmarks covering horizontal, multi-oriented, and curved text, in terms of speed and accuracy.
Takeaways & Limitations
Integrating binarization into training and adaptively fusing scales improves segmentation-based scene text detection accuracy while preserving an efficient pipeline.
Takeaways & Limitations
DBNet++ has difficulty detecting a text instance located exactly in the center region of another text instance.
Abstract
from arXiv · showhide
Recently, segmentation-based scene text detection methods have drawn extensive attention in the scene text detection field, because of their superiority in detecting the text instances of arbitrary shapes and extreme aspect ratios, profiting from the pixel-level descriptions. However, the vast majority of the existing segmentation-based approaches are limited to their complex post-processing algorithms and the scale robustness of their segmentation models, where the post-processing algorithms are not only isolated to the model optimization but also time-consuming and the scale robustness is usually strengthened by fusing multi-scale feature maps directly. In this paper, we propose a Differentiable Binarization (DB) module that integrates the binarization process, one of the most important steps in the post-processing procedure, into a segmentation network. Optimized along with the proposed DB module, the segmentation network can produce more accurate results, which enhances the accuracy of text detection with a simple pipeline. Furthermore, an efficient Adaptive Scale Fusion (ASF) module is proposed to improve the scale robustness by fusing features of different scales adaptively. By incorporating the proposed DB and ASF with the segmentation network, our proposed scene text detector consistently achieves state-of-the-art results, in terms of both detection accuracy and speed, on five standard benchmarks.
1 INTRODUCTION
Scene text detection remains difficult across diverse scales, irregular shapes, and extreme aspect ratios, while segmentation-based systems often incur costly post-processing and insufficient scale adaptability. DBNet++ addresses these issues with differentiable binarization and adaptive scale fusion, achieving strong accuracy and efficiency across benchmarks.
- Motivation: Scene text detection is challenging because text instances vary in scale, shape, and aspect ratio.These capabilities matter for applications including office automation, visual search, geo-location, and blind auxiliary.
- Motivation: Segmentation-based detectors handle irregular shapes and extreme aspect ratios but commonly depend on complex, time-consuming post-processing.Traditional pipelines threshold probability maps and then use heuristic pixel grouping, with these operations isolated from network training.
- Method: Differentiable Binarization inserts an approximate binarization operation and an adaptively predicted threshold map into the segmentation network for joint optimization.The threshold map accommodates differing confidence levels across text regions, including lower-confidence boundaries.
- Method: Adaptive Scale Fusion combines stage-wise and spatial attention to adaptively fuse multi-scale features and improve robustness across text-instance scales.Unlike direct multi-scale fusion, ASF learns scale-specific and spatially varying attention weights.
- Contributions: DBNet++ integrates DB and ASF into a segmentation network, simplifying post-processing while strengthening segmentation quality and scale robustness.The proposed detector is described as accurate, robust, and efficient.
- Contributions: DBNet++ achieves consistently state-of-the-art accuracy on five benchmarks covering horizontal, multi-oriented, and curved text.The method also removes DB during inference without sacrificing accuracy or adding inference memory/time cost.
2 RELATED WORK
Related work spans regression-based, part-based, and segmentation-based scene text detectors, alongside efforts to improve speed and multi-scale representation. Existing approaches use varied box, component, linking, segmentation, attention, and feature-fusion strategies, with some relying on complex or difficult-to-tune procedures.
- Text Detection: Deep-learning scene text detectors are commonly categorized as regression-based, part-based, or segmentation-based according to prediction granularity.These categories distinguish direct box regression, component linking, and pixel-level prediction pipelines.
- Text Detection: Regression-based methods directly regress text bounding boxes, including quadrilateral regression and attention-based region identification.Examples include TextBoxes, TextBoxes++, DMPNet, SSTD, and RRD.
- Text Detection: Part-based methods detect text components and link them into word or text-line boxes, but their linking algorithms can be complex and difficult to tune.SegLink and SegLink++ target long or closely spaced text instances through component links and grouping.
- Text Detection: Segmentation-based methods combine pixel-level prediction with post-processing to produce bounding boxes for multi-oriented and arbitrary-shape text.Examples use semantic segmentation, text borders, instance segmentation, progressive scale expansion, or pixel embedding.
- Fast Scene Text Detection: Fast detectors prioritize accuracy and inference speed, but many earlier methods struggle with irregular text shapes.EAST uses an anchor-free design for a favorable speed-accuracy tradeoff, while the cited fast detectors generally have limitations on curved text.
- Attention Mechanisms: Attention mechanisms from image classification provide precedents for recalibrating feature responses across channels and spatial dimensions.Residual attention and squeeze-and-excitation blocks are representative examples.
- Multi-Scale Feature Fusion: Multi-scale feature fusion is widely used in semantic segmentation because context and scale are closely related.Methods such as FCN, U-Net, PSPNet, DeepLabv3, RefineNet, and related models fuse features across resolutions.
3 METHODOLOGY
DBNet++ combines differentiable binarization, adaptive thresholding, and adaptive scale fusion within a segmentation network. These components make binarization trainable, emphasize ambiguous boundaries, and adaptively combine features across scales.
- Differentiable Binarization: The network predicts probability and threshold maps, then computes an approximate binary map with a differentiable binarization function.The adaptive threshold map T and amplifying factor k define the approximate binary map, which can be optimized jointly with the segmentation network.
- Adaptive Scale Fusion: ASF adaptively fuses multi-scale features using stage-wise and spatial attention to improve scale robustness.Features are resized to a common resolution, and learned spatial weights modulate the corresponding feature maps before fusion.
- Differentiable Binarization: Differentiable binarization enlarges backpropagation feedback near ambiguous boundaries, encouraging more distinctive predictions around the decision boundary.The amplifying factor k increases derivative magnitude near the boundary, while the Sigmoid limits extreme derivative behavior.
4 EXPERIMENTS
Experiments evaluate DBNet++ across datasets, ablations, and comparisons with prior methods. The results show gains in accuracy, scale robustness, and speed across varied text shapes, orientations, languages, and scales, with a stated limitation on nested text.
- Ablation Study: 3.7% and 4.9% F-measure gains are reported for DB with ResNet-18 on MSRA-TD500 and CTW1500, respectively.With ResNet-50, DB improves F-measure by 3.2% and 4.6% on the same datasets, while remaining removable during inference without speed loss.
- Comparisons with Previous Methods: DBNet++ outperforms previous state-of-the-art methods by 1.0% and 1.6% on Total-Text and CTW1500, respectively.The paper reports state-of-the-art accuracy and speed on curved-text benchmarks, while ResNet-18 offers faster inference with a small performance drop.
- Comparisons with Previous Methods: DBNet++ surpasses previous state-of-the-art accuracy by 3.1% on MSRA-TD500 and 3.3% on MLT-2019.DBNet++ with ResNet-18 at 512 input size reaches 80 FPS, while the 736 setting is faster than the previous fastest method on MSRA-TD500 with better accuracy.
- Conference-Version Comparison: DBNet++ improves F-measure over DBNet by 0.5%, 2.9%, 0.8%, 3.6%, and 1.3% on Total-Text, CTW1500, ICDAR 2015, MSRA-TD500, and MLT-2019 with ResNet-18.The corresponding ResNet-50 changes are 1.3%, 1.9%, 0.0%, 2.3%, and 1.0%, with little speed drop.
5 CONCLUSION
The paper presents DBNet++ for arbitrary-shape scene text detection by integrating differentiable binarization and adaptive scale fusion. Experiments report consistent state-of-the-art speed and accuracy, including real-time performance with ResNet-18.
- Conclusion: DB integrates binarization into training, while ASF enhances the segmentation network’s scale robustness.Together, the modules improve segmentation-based scene text detection from two complementary aspects.
- Conclusion: DBNet++ consistently outperforms state-of-the-art methods on five standard benchmarks in speed and accuracy with a ResNet-50 backbone.The benchmarks cover arbitrary-shape scene text detection settings.
- Conclusion: ResNet-18 provides competitive performance on all testing datasets with real-time inference speed.This supports a lightweight deployment configuration within the reported experimental scope.
1. Qualitative Comparisons between DBNet and DBNet++
The supplied passages identify the qualitative comparison columns as GroundTruth, DBNet, and DBNet++.
- GroundTruth is one of the three labels in the qualitative comparison.
- DBNet is one of the three labels in the qualitative comparison.
- DBNet++ is one of the three labels in the qualitative comparison.